Merge branch 'develop' into lua-ref-target
lethosor
commit
a44a2e2298
@ -0,0 +1,6 @@
|
|||||||
|
Thank you for your interest in contributing to DFHack! If you're reading this
|
||||||
|
document, you're probably viewing it on GitHub. The DFHack docs are hosted
|
||||||
|
on [ReadTheDocs](https://dfhack.readthedocs.io/) - in particular, contributing
|
||||||
|
guidelines are [here](https://docs.dfhack.org/en/latest/docs/Contributing.html).
|
||||||
|
Double-checking the style guidelines before submitting a pull request is
|
||||||
|
always appreciated.
|
||||||
@ -1,229 +0,0 @@
|
|||||||
###########################
|
|
||||||
How to contribute to DFHack
|
|
||||||
###########################
|
|
||||||
|
|
||||||
.. contents::
|
|
||||||
|
|
||||||
|
|
||||||
.. _contributing-code:
|
|
||||||
|
|
||||||
Contributing Code
|
|
||||||
=================
|
|
||||||
Several things should be kept in mind when contributing code to DFHack.
|
|
||||||
|
|
||||||
Code Format
|
|
||||||
-----------
|
|
||||||
* Four space indents for C++. Never use tabs for indentation in any language.
|
|
||||||
* LF (Unix style) line terminators
|
|
||||||
* Avoid trailing whitespace
|
|
||||||
* UTF-8 encoding
|
|
||||||
* For C++:
|
|
||||||
|
|
||||||
* Opening and closing braces on their own lines or opening brace at the end of the previous line
|
|
||||||
* Braces placed at original indent level if on their own lines
|
|
||||||
* #includes should be sorted. C++ libraries first, then dfhack modules, then df structures,
|
|
||||||
then local includes. Within each category they should be sorted alphabetically.
|
|
||||||
|
|
||||||
How to get new code into DFHack
|
|
||||||
-------------------------------
|
|
||||||
* Submit pull requests to the ``develop`` branch, not the ``master`` branch.
|
|
||||||
(The ``master`` branch always points at the most recent release)
|
|
||||||
* Use a new branch for each feature or bugfix so that your changes can be merged independently
|
|
||||||
(i.e. not the master or develop branch of your fork).
|
|
||||||
* If possible, compile on multiple platforms when changing anything that compiles
|
|
||||||
* It must pass CI - run ``python travis/all.py`` to check this.
|
|
||||||
* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See
|
|
||||||
`build-changelog` for more information on the changelog format.
|
|
||||||
* Create a GitHub pull request once finished
|
|
||||||
* Submit ideas and bug reports as :issue:`issues on GitHub <>`.
|
|
||||||
Posts in the forum thread can easily get missed or forgotten.
|
|
||||||
* Work on :issue:`reported problems <?q=is:open+-label:idea>`
|
|
||||||
will take priority over ideas or suggestions.
|
|
||||||
|
|
||||||
.. _contributing-memory-research:
|
|
||||||
|
|
||||||
Memory research
|
|
||||||
---------------
|
|
||||||
If you want to do memory research, you'll need some tools and some knowledge.
|
|
||||||
In general, you'll need a good memory viewer and optionally something
|
|
||||||
to look at machine code without getting crazy :)
|
|
||||||
Using publicly known information and analyzing the game's data is preferred.
|
|
||||||
|
|
||||||
Good Windows tools include:
|
|
||||||
|
|
||||||
* IDA Freeware 7.0 (for non-commercial use, supports 32-bit and 64-bit)
|
|
||||||
* Cheat Engine
|
|
||||||
|
|
||||||
Good Linux tools:
|
|
||||||
|
|
||||||
* angavrilov's df-structures gui (32-bit only, visit us on IRC for details)
|
|
||||||
* IDA Freeware 7.0 (see above)
|
|
||||||
* edb (Evan's Debugger)
|
|
||||||
* Some of the tools residing in the ``legacy`` dfhack branch.
|
|
||||||
|
|
||||||
Using the library as a developer
|
|
||||||
================================
|
|
||||||
Currently, the most direct way to use the library is to write a script or plugin that can be loaded by it.
|
|
||||||
All the plugins can be found in the 'plugins' folder. There's no in-depth documentation
|
|
||||||
on how to write one yet, but it should be easy enough to copy one and just follow the pattern.
|
|
||||||
``plugins/skeleton/skeleton.cpp`` is provided for this purpose.
|
|
||||||
|
|
||||||
Other than through plugins, it is possible to use DFHack via remote access interface,
|
|
||||||
or by writing scripts in Lua or Ruby. There are plenty of examples in the scripts folder.
|
|
||||||
The `lua-api` is quite well documented.
|
|
||||||
|
|
||||||
The most important parts of DFHack are the Core, Console, Modules and Plugins.
|
|
||||||
|
|
||||||
* Core acts as the centerpiece of DFHack - it acts as a filter between DF and
|
|
||||||
SDL and synchronizes the various plugins with DF.
|
|
||||||
* Console is a thread-safe console that can be used to invoke commands exported by Plugins.
|
|
||||||
* Modules actually describe the way to access information in DF's memory. You
|
|
||||||
can get them from the Core. Most modules are split into two parts: high-level
|
|
||||||
and low-level. High-level is mostly method calls, low-level publicly visible
|
|
||||||
pointers to DF's data structures.
|
|
||||||
* Plugins are the tools that use all the other stuff to make things happen.
|
|
||||||
A plugin can have a list of commands that it exports and an onupdate function
|
|
||||||
that will be called each DF game tick.
|
|
||||||
|
|
||||||
Rudimentary API documentation can be built using doxygen (see build options
|
|
||||||
in ``CMakeCache.txt`` or with ``ccmake`` or ``cmake-gui``). The full DFHack
|
|
||||||
documentation is built with Sphinx_, which runs automatically at compile time.
|
|
||||||
|
|
||||||
.. _Sphinx: http://www.sphinx-doc.org
|
|
||||||
|
|
||||||
DFHack consists of variously licensed code, but invariably weak copyleft.
|
|
||||||
The main license is zlib/libpng, some bits are MIT licensed, and some are
|
|
||||||
BSD licensed. See the `license` for more information.
|
|
||||||
|
|
||||||
Feel free to add your own extensions and plugins. Contributing back to
|
|
||||||
the DFHack repository is welcome and the right thing to do :)
|
|
||||||
|
|
||||||
DF data structure definitions
|
|
||||||
-----------------------------
|
|
||||||
DFHack uses information about the game data structures, represented via xml files
|
|
||||||
in the ``library/xml/`` submodule.
|
|
||||||
|
|
||||||
See https://github.com/DFHack/df-structures, and the documentation linked in the index.
|
|
||||||
|
|
||||||
Data structure layouts are described in files following the ``df.*.xml`` name pattern.
|
|
||||||
This information is transformed by a perl script into C++ headers describing the
|
|
||||||
structures, and associated metadata for the Lua wrapper. These headers and data
|
|
||||||
are then compiled into the DFHack libraries, thus necessitating a compatibility
|
|
||||||
break every time layouts change; in return it significantly boosts the efficiency
|
|
||||||
and capabilities of DFHack code.
|
|
||||||
|
|
||||||
Global object addresses are stored in :file:`symbols.xml`, which is copied to the dfhack
|
|
||||||
release package and loaded as data at runtime.
|
|
||||||
|
|
||||||
Remote access interface
|
|
||||||
-----------------------
|
|
||||||
DFHack supports remote access by exchanging Google protobuf messages via a TCP
|
|
||||||
socket. Both the core and plugins can define remotely accessible methods. The
|
|
||||||
``dfhack-run`` command uses this interface to invoke ordinary console commands.
|
|
||||||
|
|
||||||
Currently the supported set of requests is limited, because the developers don't
|
|
||||||
know what exactly is most useful. `remotefortressreader` provides a fairly
|
|
||||||
comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`.
|
|
||||||
|
|
||||||
|
|
||||||
Documentation Standards
|
|
||||||
=======================
|
|
||||||
DFHack documentation is built with Sphinx_, and configured automatically
|
|
||||||
through CMake. If you want to build the docs *only*, use this command::
|
|
||||||
|
|
||||||
sphinx-build . docs/html
|
|
||||||
|
|
||||||
Whether you're adding new code or just fixing old documentation (and there's plenty),
|
|
||||||
there are a few important standards for completeness and consistent style. Treat
|
|
||||||
this section as a guide rather than iron law, match the surrounding text, and you'll
|
|
||||||
be fine.
|
|
||||||
|
|
||||||
Each command should have a short (~54 character) help string, which is shown
|
|
||||||
by the `ls` command. For scripts, this is a comment on the first line
|
|
||||||
(the comment marker and whitespace is stripped). For plugins it's the second
|
|
||||||
argument to ``PluginCommand``. Please make this brief but descriptive!
|
|
||||||
|
|
||||||
Everything should be documented! If it's not clear *where* a particular
|
|
||||||
thing should be documented, ask on IRC or in the DFHack thread on Bay12 -
|
|
||||||
as well as getting help, you'll be providing valuable feedback that
|
|
||||||
makes it easier for future readers!
|
|
||||||
|
|
||||||
Scripts can use a custom autodoc function, based on the Sphinx ``include``
|
|
||||||
directive - anything between the tokens is copied into the appropriate scripts
|
|
||||||
documentation page. For Ruby, we follow the built-in docstring convention
|
|
||||||
(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]``
|
|
||||||
- ordinary multi-line strings. It is highly encouraged to reuse this string
|
|
||||||
as the in-console documentation by (eg.) printing it when a ``-help`` argument
|
|
||||||
is given.
|
|
||||||
|
|
||||||
The docs **must** have a heading which exactly matches the command, underlined
|
|
||||||
with ``=====`` to the same length. For example, a lua file would have::
|
|
||||||
|
|
||||||
local helpstr = [====[
|
|
||||||
|
|
||||||
add-thought
|
|
||||||
===========
|
|
||||||
Adds a thought or emotion to the selected unit. Can be used by other scripts,
|
|
||||||
or the gui invoked by running ``add-thought gui`` with a unit selected.
|
|
||||||
|
|
||||||
]====]
|
|
||||||
|
|
||||||
|
|
||||||
Where the heading for a section is also the name of a command, the spelling
|
|
||||||
and case should exactly match the command to enter in the DFHack command line.
|
|
||||||
|
|
||||||
Try to keep lines within 80-100 characters, so it's readable in plain text
|
|
||||||
in the terminal - Sphinx (our documentation system) will make sure
|
|
||||||
paragraphs flow.
|
|
||||||
|
|
||||||
If there aren't many options or examples to show, they can go in a paragraph of
|
|
||||||
text. Use double-backticks to put commands in monospaced font, like this::
|
|
||||||
|
|
||||||
You can use ``cleanowned scattered x`` to dump tattered or abandoned items.
|
|
||||||
|
|
||||||
If the command takes more than three arguments, format the list as a table
|
|
||||||
called Usage. The table *only* lists arguments, not full commands.
|
|
||||||
Input values are specified in angle brackets. Example::
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
|
|
||||||
:arg1: A simple argument.
|
|
||||||
:arg2 <input>: Does something based on the input value.
|
|
||||||
:Very long argument:
|
|
||||||
Is very specific.
|
|
||||||
|
|
||||||
To demonstrate usage - useful mainly when the syntax is complicated, list the
|
|
||||||
full command with arguments in monospaced font, then indent the next line and
|
|
||||||
describe the effect::
|
|
||||||
|
|
||||||
``resume all``
|
|
||||||
Resumes all suspended constructions.
|
|
||||||
|
|
||||||
If it would be helpful to mention another DFHack command, don't just type the
|
|
||||||
name - add a hyperlink! Specify the link target in backticks, and it will be
|
|
||||||
replaced with the corresponding title and linked: eg ```autolabor```
|
|
||||||
=> `autolabor`. Link targets should be equivalent to the command
|
|
||||||
described (without file extension), and placed above the heading of that
|
|
||||||
section like this::
|
|
||||||
|
|
||||||
.. _autolabor:
|
|
||||||
|
|
||||||
autolabor
|
|
||||||
=========
|
|
||||||
|
|
||||||
Add link targets if you need them, but otherwise plain headings are preferred.
|
|
||||||
Scripts have link targets created automatically.
|
|
||||||
|
|
||||||
Other ways to help
|
|
||||||
==================
|
|
||||||
DFHack is a software project, but there's a lot more to it than programming.
|
|
||||||
If you're not comfortable programming, you can help by:
|
|
||||||
|
|
||||||
* reporting bugs and incomplete documentation
|
|
||||||
* improving the documentation
|
|
||||||
* finding third-party scripts to add
|
|
||||||
* writing tutorials for newbies
|
|
||||||
|
|
||||||
All those things are crucial, and often under-represented. So if that's
|
|
||||||
your thing, go get started!
|
|
||||||
|
|
||||||
@ -1,249 +0,0 @@
|
|||||||
/* -*- mode: c++; tab-width: 2; indent-tabs-mode: nil; -*-
|
|
||||||
Copyright (c) 2010-2012 Marcus Geelnard
|
|
||||||
|
|
||||||
This software is provided 'as-is', without any express or implied
|
|
||||||
warranty. In no event will the authors be held liable for any damages
|
|
||||||
arising from the use of this software.
|
|
||||||
|
|
||||||
Permission is granted to anyone to use this software for any purpose,
|
|
||||||
including commercial applications, and to alter it and redistribute it
|
|
||||||
freely, subject to the following restrictions:
|
|
||||||
|
|
||||||
1. The origin of this software must not be misrepresented; you must not
|
|
||||||
claim that you wrote the original software. If you use this software
|
|
||||||
in a product, an acknowledgment in the product documentation would be
|
|
||||||
appreciated but is not required.
|
|
||||||
|
|
||||||
2. Altered source versions must be plainly marked as such, and must not be
|
|
||||||
misrepresented as being the original software.
|
|
||||||
|
|
||||||
3. This notice may not be removed or altered from any source
|
|
||||||
distribution.
|
|
||||||
*/
|
|
||||||
|
|
||||||
#ifndef _FAST_MUTEX_H_
|
|
||||||
#define _FAST_MUTEX_H_
|
|
||||||
|
|
||||||
/// @file
|
|
||||||
|
|
||||||
// Which platform are we on?
|
|
||||||
#if !defined(_TTHREAD_PLATFORM_DEFINED_)
|
|
||||||
#if defined(_WIN32) || defined(__WIN32__) || defined(__WINDOWS__)
|
|
||||||
#define _TTHREAD_WIN32_
|
|
||||||
#else
|
|
||||||
#define _TTHREAD_POSIX_
|
|
||||||
#endif
|
|
||||||
#define _TTHREAD_PLATFORM_DEFINED_
|
|
||||||
#endif
|
|
||||||
|
|
||||||
// Check if we can support the assembly language level implementation (otherwise
|
|
||||||
// revert to the system API)
|
|
||||||
#if (defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__))) || \
|
|
||||||
(defined(_MSC_VER) && (defined(_M_IX86) /*|| defined(_M_X64)*/)) || \
|
|
||||||
(defined(__GNUC__) && (defined(__ppc__)))
|
|
||||||
#define _FAST_MUTEX_ASM_
|
|
||||||
#else
|
|
||||||
#define _FAST_MUTEX_SYS_
|
|
||||||
#endif
|
|
||||||
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
#define NOMINMAX
|
|
||||||
#ifndef WIN32_LEAN_AND_MEAN
|
|
||||||
#define WIN32_LEAN_AND_MEAN
|
|
||||||
#define __UNDEF_LEAN_AND_MEAN
|
|
||||||
#endif
|
|
||||||
#include <windows.h>
|
|
||||||
#ifdef __UNDEF_LEAN_AND_MEAN
|
|
||||||
#undef WIN32_LEAN_AND_MEAN
|
|
||||||
#undef __UNDEF_LEAN_AND_MEAN
|
|
||||||
#endif
|
|
||||||
#else
|
|
||||||
#ifdef _FAST_MUTEX_ASM_
|
|
||||||
#include <sched.h>
|
|
||||||
#else
|
|
||||||
#include <pthread.h>
|
|
||||||
#endif
|
|
||||||
#endif
|
|
||||||
|
|
||||||
namespace tthread {
|
|
||||||
|
|
||||||
/// Fast mutex class.
|
|
||||||
/// This is a mutual exclusion object for synchronizing access to shared
|
|
||||||
/// memory areas for several threads. It is similar to the tthread::mutex class,
|
|
||||||
/// but instead of using system level functions, it is implemented as an atomic
|
|
||||||
/// spin lock with very low CPU overhead.
|
|
||||||
///
|
|
||||||
/// The \c fast_mutex class is NOT compatible with the \c condition_variable
|
|
||||||
/// class (however, it IS compatible with the \c lock_guard class). It should
|
|
||||||
/// also be noted that the \c fast_mutex class typically does not provide
|
|
||||||
/// as accurate thread scheduling as a the standard \c mutex class does.
|
|
||||||
///
|
|
||||||
/// Because of the limitations of the class, it should only be used in
|
|
||||||
/// situations where the mutex needs to be locked/unlocked very frequently.
|
|
||||||
///
|
|
||||||
/// @note The "fast" version of this class relies on inline assembler language,
|
|
||||||
/// which is currently only supported for 32/64-bit Intel x86/AMD64 and
|
|
||||||
/// PowerPC architectures on a limited number of compilers (GNU g++ and MS
|
|
||||||
/// Visual C++).
|
|
||||||
/// For other architectures/compilers, system functions are used instead.
|
|
||||||
class fast_mutex {
|
|
||||||
public:
|
|
||||||
/// Constructor.
|
|
||||||
#if defined(_FAST_MUTEX_ASM_)
|
|
||||||
fast_mutex() : mLock(0) {}
|
|
||||||
#else
|
|
||||||
fast_mutex()
|
|
||||||
{
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
InitializeCriticalSection(&mHandle);
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
pthread_mutex_init(&mHandle, NULL);
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
#endif
|
|
||||||
|
|
||||||
#if !defined(_FAST_MUTEX_ASM_)
|
|
||||||
/// Destructor.
|
|
||||||
~fast_mutex()
|
|
||||||
{
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
DeleteCriticalSection(&mHandle);
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
pthread_mutex_destroy(&mHandle);
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
#endif
|
|
||||||
|
|
||||||
/// Lock the mutex.
|
|
||||||
/// The method will block the calling thread until a lock on the mutex can
|
|
||||||
/// be obtained. The mutex remains locked until \c unlock() is called.
|
|
||||||
/// @see lock_guard
|
|
||||||
inline void lock()
|
|
||||||
{
|
|
||||||
#if defined(_FAST_MUTEX_ASM_)
|
|
||||||
bool gotLock;
|
|
||||||
do {
|
|
||||||
gotLock = try_lock();
|
|
||||||
if(!gotLock)
|
|
||||||
{
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
Sleep(0);
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
sched_yield();
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
} while(!gotLock);
|
|
||||||
#else
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
EnterCriticalSection(&mHandle);
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
pthread_mutex_lock(&mHandle);
|
|
||||||
#endif
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Try to lock the mutex.
|
|
||||||
/// The method will try to lock the mutex. If it fails, the function will
|
|
||||||
/// return immediately (non-blocking).
|
|
||||||
/// @return \c true if the lock was acquired, or \c false if the lock could
|
|
||||||
/// not be acquired.
|
|
||||||
inline bool try_lock()
|
|
||||||
{
|
|
||||||
#if defined(_FAST_MUTEX_ASM_)
|
|
||||||
int oldLock;
|
|
||||||
#if defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__))
|
|
||||||
asm volatile (
|
|
||||||
"movl $1,%%eax\n\t"
|
|
||||||
"xchg %%eax,%0\n\t"
|
|
||||||
"movl %%eax,%1\n\t"
|
|
||||||
: "=m" (mLock), "=m" (oldLock)
|
|
||||||
:
|
|
||||||
: "%eax", "memory"
|
|
||||||
);
|
|
||||||
#elif defined(_MSC_VER) && (defined(_M_IX86) || defined(_M_X64))
|
|
||||||
int *ptrLock = &mLock;
|
|
||||||
__asm {
|
|
||||||
mov eax,1
|
|
||||||
mov ecx,ptrLock
|
|
||||||
xchg eax,[ecx]
|
|
||||||
mov oldLock,eax
|
|
||||||
}
|
|
||||||
#elif defined(__GNUC__) && (defined(__ppc__))
|
|
||||||
int newLock = 1;
|
|
||||||
asm volatile (
|
|
||||||
"\n1:\n\t"
|
|
||||||
"lwarx %0,0,%1\n\t"
|
|
||||||
"cmpwi 0,%0,0\n\t"
|
|
||||||
"bne- 2f\n\t"
|
|
||||||
"stwcx. %2,0,%1\n\t"
|
|
||||||
"bne- 1b\n\t"
|
|
||||||
"isync\n"
|
|
||||||
"2:\n\t"
|
|
||||||
: "=&r" (oldLock)
|
|
||||||
: "r" (&mLock), "r" (newLock)
|
|
||||||
: "cr0", "memory"
|
|
||||||
);
|
|
||||||
#endif
|
|
||||||
return (oldLock == 0);
|
|
||||||
#else
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
return TryEnterCriticalSection(&mHandle) ? true : false;
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
return (pthread_mutex_trylock(&mHandle) == 0) ? true : false;
|
|
||||||
#endif
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Unlock the mutex.
|
|
||||||
/// If any threads are waiting for the lock on this mutex, one of them will
|
|
||||||
/// be unblocked.
|
|
||||||
inline void unlock()
|
|
||||||
{
|
|
||||||
#if defined(_FAST_MUTEX_ASM_)
|
|
||||||
#if defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__))
|
|
||||||
asm volatile (
|
|
||||||
"movl $0,%%eax\n\t"
|
|
||||||
"xchg %%eax,%0\n\t"
|
|
||||||
: "=m" (mLock)
|
|
||||||
:
|
|
||||||
: "%eax", "memory"
|
|
||||||
);
|
|
||||||
#elif defined(_MSC_VER) && (defined(_M_IX86) || defined(_M_X64))
|
|
||||||
int *ptrLock = &mLock;
|
|
||||||
__asm {
|
|
||||||
mov eax,0
|
|
||||||
mov ecx,ptrLock
|
|
||||||
xchg eax,[ecx]
|
|
||||||
}
|
|
||||||
#elif defined(__GNUC__) && (defined(__ppc__))
|
|
||||||
asm volatile (
|
|
||||||
"sync\n\t" // Replace with lwsync where possible?
|
|
||||||
: : : "memory"
|
|
||||||
);
|
|
||||||
mLock = 0;
|
|
||||||
#endif
|
|
||||||
#else
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
LeaveCriticalSection(&mHandle);
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
pthread_mutex_unlock(&mHandle);
|
|
||||||
#endif
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
|
|
||||||
private:
|
|
||||||
#if defined(_FAST_MUTEX_ASM_)
|
|
||||||
int mLock;
|
|
||||||
#else
|
|
||||||
#if defined(_TTHREAD_WIN32_)
|
|
||||||
CRITICAL_SECTION mHandle;
|
|
||||||
#elif defined(_TTHREAD_POSIX_)
|
|
||||||
pthread_mutex_t mHandle;
|
|
||||||
#endif
|
|
||||||
#endif
|
|
||||||
};
|
|
||||||
|
|
||||||
}
|
|
||||||
|
|
||||||
#endif // _FAST_MUTEX_H_
|
|
||||||
|
|
||||||
@ -0,0 +1,59 @@
|
|||||||
|
.. _contributing:
|
||||||
|
|
||||||
|
###########################
|
||||||
|
How to contribute to DFHack
|
||||||
|
###########################
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
.. _contributing-code:
|
||||||
|
|
||||||
|
Contributing Code
|
||||||
|
=================
|
||||||
|
Several things should be kept in mind when contributing code to DFHack.
|
||||||
|
|
||||||
|
Code Format
|
||||||
|
-----------
|
||||||
|
* Four space indents for C++. Never use tabs for indentation in any language.
|
||||||
|
* LF (Unix style) line terminators
|
||||||
|
* Avoid trailing whitespace
|
||||||
|
* UTF-8 encoding
|
||||||
|
* For C++:
|
||||||
|
|
||||||
|
* Opening and closing braces on their own lines or opening brace at the end of the previous line
|
||||||
|
* Braces placed at original indent level if on their own lines
|
||||||
|
* #includes should be sorted. C++ libraries first, then dfhack modules, then df structures,
|
||||||
|
then local includes. Within each category they should be sorted alphabetically.
|
||||||
|
|
||||||
|
How to get new code into DFHack
|
||||||
|
-------------------------------
|
||||||
|
* Submit pull requests to the ``develop`` branch, not the ``master`` branch.
|
||||||
|
(The ``master`` branch always points at the most recent release)
|
||||||
|
* Use a new branch for each feature or bugfix so that your changes can be merged independently
|
||||||
|
(i.e. not the master or develop branch of your fork).
|
||||||
|
* If possible, compile on multiple platforms when changing anything that compiles
|
||||||
|
* It must pass CI - run ``python travis/all.py`` to check this.
|
||||||
|
* Update documentation when applicable - see `docs-standards` for details.
|
||||||
|
* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See
|
||||||
|
`build-changelog` for more information on the changelog format.
|
||||||
|
* Create a GitHub pull request once finished
|
||||||
|
* Submit ideas and bug reports as :issue:`issues on GitHub <>`.
|
||||||
|
Posts in the forum thread can easily get missed or forgotten.
|
||||||
|
* Work on :issue:`reported problems <?q=is:open+-label:idea>`
|
||||||
|
will take priority over ideas or suggestions.
|
||||||
|
|
||||||
|
|
||||||
|
Other ways to help
|
||||||
|
==================
|
||||||
|
DFHack is a software project, but there's a lot more to it than programming.
|
||||||
|
If you're not comfortable programming, you can help by:
|
||||||
|
|
||||||
|
* reporting bugs and incomplete documentation
|
||||||
|
* improving the documentation
|
||||||
|
* finding third-party scripts to add
|
||||||
|
* writing tutorials for newbies
|
||||||
|
|
||||||
|
All those things are crucial, and often under-represented. So if that's
|
||||||
|
your thing, go get started!
|
||||||
|
|
||||||
@ -0,0 +1,87 @@
|
|||||||
|
===========================
|
||||||
|
DFHack development overview
|
||||||
|
===========================
|
||||||
|
|
||||||
|
DFHack has various components; this page provides an overview of some. If you
|
||||||
|
are looking to develop a tool for DFHack, developing a script or plugin is
|
||||||
|
likely the most straightforward choice.
|
||||||
|
|
||||||
|
Other pages that may be relevant include:
|
||||||
|
|
||||||
|
- `contributing`
|
||||||
|
- `documentation`
|
||||||
|
- `license`
|
||||||
|
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
|
||||||
|
Plugins
|
||||||
|
-------
|
||||||
|
|
||||||
|
DFHack plugins are written in C++ and located in the ``plugins`` folder.
|
||||||
|
Currently, documentation on how to write plugins is somewhat sparse. There are
|
||||||
|
templates that you can get use to get started in the ``plugins/skeleton``
|
||||||
|
folder, and the source code of existing plugins can also be helpful.
|
||||||
|
|
||||||
|
If you want to compile a plugin that you have just added, you will need to add a
|
||||||
|
call to ``DFHACK_PLUGIN`` in ``plugins/CMakeLists.txt``.
|
||||||
|
|
||||||
|
Plugins have the ability to make one or more commands available to users of the
|
||||||
|
DFHack console. Examples include `3dveins` (which implements the ``3dveins``
|
||||||
|
command) and `reveal` (which implements ``reveal``, ``unreveal``, and several
|
||||||
|
other commands).
|
||||||
|
|
||||||
|
Plugins can also register handlers to run on every tick, and can interface with
|
||||||
|
the built-in `enable` and `disable` commands. For the full plugin API, see the
|
||||||
|
skeleton plugins or ``PluginManager.cpp``.
|
||||||
|
|
||||||
|
Installed plugins live in the ``hack/plugins`` folder of a DFHack installation,
|
||||||
|
and the `load` family of commands can be used to load a recompiled plugin
|
||||||
|
without restarting DF.
|
||||||
|
|
||||||
|
See `plugins-index` for a list of all plugins included in DFHack.
|
||||||
|
|
||||||
|
Scripts
|
||||||
|
-------
|
||||||
|
|
||||||
|
DFHack scripts can currently be written in Lua or Ruby. The `Lua API <lua-api>`
|
||||||
|
is more complete and currently better-documented, however. Referring to existing
|
||||||
|
scripts as well as the API documentation can be helpful when developing new
|
||||||
|
scripts.
|
||||||
|
|
||||||
|
`Scripts included in DFHack <scripts-index>` live in a separate `scripts repository <https://github.com/dfhack/scripts>`_.
|
||||||
|
This can be found in the ``scripts`` submodule if you have
|
||||||
|
`cloned DFHack <compile-how-to-get-the-code>`, or the ``hack/scripts`` folder
|
||||||
|
of an installed copy of DFHack.
|
||||||
|
|
||||||
|
Core
|
||||||
|
----
|
||||||
|
|
||||||
|
The `DFHack core <dfhack-core>` has a variety of low-level functions. It is
|
||||||
|
responsible for hooking into DF (via SDL), providing a console, and providing an
|
||||||
|
interface for plugins and scripts to interact with DF.
|
||||||
|
|
||||||
|
Modules
|
||||||
|
-------
|
||||||
|
|
||||||
|
A lot of shared code to interact with DF in more complicated ways is contained
|
||||||
|
in **modules**. For example, the Units module contains functions for checking
|
||||||
|
various traits of units, changing nicknames properly, and more. Generally, code
|
||||||
|
that is useful to multiple plugins and scripts should go in the appropriate
|
||||||
|
module, if there is one.
|
||||||
|
|
||||||
|
Several modules are also `exposed to Lua <lua-cpp-func-wrappers>`, although
|
||||||
|
some functions (and some entire modules) are currently only available in C++.
|
||||||
|
|
||||||
|
Remote access interface
|
||||||
|
-----------------------
|
||||||
|
DFHack supports remote access by exchanging Google protobuf messages via a TCP
|
||||||
|
socket. Both the core and plugins can define remotely accessible methods. The
|
||||||
|
``dfhack-run`` command uses this interface to invoke ordinary console commands.
|
||||||
|
|
||||||
|
Currently the supported set of requests is limited, because the developers don't
|
||||||
|
know what exactly is most useful. `remotefortressreader` provides a fairly
|
||||||
|
comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`.
|
||||||
|
|
||||||
@ -0,0 +1,285 @@
|
|||||||
|
.. _documentation:
|
||||||
|
|
||||||
|
###########################
|
||||||
|
DFHack Documentation System
|
||||||
|
###########################
|
||||||
|
|
||||||
|
|
||||||
|
DFHack documentation, like the file you are reading now, is created as ``.rst`` files,
|
||||||
|
which are in `reStructuredText (reST) <http://sphinx-doc.org/rest.html>`_ format.
|
||||||
|
This is a documentation format common in the Python community. It is very
|
||||||
|
similar in concept - and in syntax - to Markdown, as found on GitHub and many other
|
||||||
|
places. However it is more advanced than Markdown, with more features available when
|
||||||
|
compiled to HTML, such as automatic tables of contents, cross-linking, special
|
||||||
|
external links (forum, wiki, etc) and more. The documentation is compiled by a
|
||||||
|
Python tool, `Sphinx <http://sphinx-doc.org>`_.
|
||||||
|
|
||||||
|
The DFHack build process will compile the documentation, but this is disabled
|
||||||
|
by default due to the additional Python and Sphinx requirements. You typically
|
||||||
|
only need to build the docs if you're changing them, or perhaps
|
||||||
|
if you want a local HTML copy; otherwise, you can read an
|
||||||
|
`online version hosted by ReadTheDocs <https://dfhack.readthedocs.org>`_.
|
||||||
|
|
||||||
|
(Note that even if you do want a local copy, it is certainly not necessary to
|
||||||
|
compile the documentation in order to read it. Like Markdown, reST documents are
|
||||||
|
designed to be just as readable in a plain-text editor as they are in HTML format.
|
||||||
|
The main thing you lose in plain text format is hyperlinking.)
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
.. _docs-standards:
|
||||||
|
|
||||||
|
Documentation standards
|
||||||
|
=======================
|
||||||
|
|
||||||
|
Whether you're adding new code or just fixing old documentation (and there's plenty),
|
||||||
|
there are a few important standards for completeness and consistent style. Treat
|
||||||
|
this section as a guide rather than iron law, match the surrounding text, and you'll
|
||||||
|
be fine.
|
||||||
|
|
||||||
|
Each command should have a short (~54 character) help string, which is shown
|
||||||
|
by the `ls` command. For scripts, this is a comment on the first line
|
||||||
|
(the comment marker and whitespace is stripped). For plugins it's the second
|
||||||
|
argument to ``PluginCommand``. Please make this brief but descriptive!
|
||||||
|
|
||||||
|
Everything should be documented! If it's not clear *where* a particular
|
||||||
|
thing should be documented, ask on IRC or in the DFHack thread on Bay12 -
|
||||||
|
as well as getting help, you'll be providing valuable feedback that
|
||||||
|
makes it easier for future readers!
|
||||||
|
|
||||||
|
Scripts can use a custom autodoc function, based on the Sphinx ``include``
|
||||||
|
directive - anything between the tokens is copied into the appropriate scripts
|
||||||
|
documentation page. For Ruby, we follow the built-in docstring convention
|
||||||
|
(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]``
|
||||||
|
- ordinary multi-line strings. It is highly encouraged to reuse this string
|
||||||
|
as the in-console documentation by (eg.) printing it when a ``-help`` argument
|
||||||
|
is given.
|
||||||
|
|
||||||
|
The docs **must** have a heading which exactly matches the command, underlined
|
||||||
|
with ``=====`` to the same length. For example, a lua file would have::
|
||||||
|
|
||||||
|
local helpstr = [====[
|
||||||
|
|
||||||
|
add-thought
|
||||||
|
===========
|
||||||
|
Adds a thought or emotion to the selected unit. Can be used by other scripts,
|
||||||
|
or the gui invoked by running ``add-thought gui`` with a unit selected.
|
||||||
|
|
||||||
|
]====]
|
||||||
|
|
||||||
|
|
||||||
|
Where the heading for a section is also the name of a command, the spelling
|
||||||
|
and case should exactly match the command to enter in the DFHack command line.
|
||||||
|
|
||||||
|
Try to keep lines within 80-100 characters, so it's readable in plain text
|
||||||
|
in the terminal - Sphinx (our documentation system) will make sure
|
||||||
|
paragraphs flow.
|
||||||
|
|
||||||
|
If there aren't many options or examples to show, they can go in a paragraph of
|
||||||
|
text. Use double-backticks to put commands in monospaced font, like this::
|
||||||
|
|
||||||
|
You can use ``cleanowned scattered x`` to dump tattered or abandoned items.
|
||||||
|
|
||||||
|
If the command takes more than three arguments, format the list as a table
|
||||||
|
called Usage. The table *only* lists arguments, not full commands.
|
||||||
|
Input values are specified in angle brackets. Example::
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
:arg1: A simple argument.
|
||||||
|
:arg2 <input>: Does something based on the input value.
|
||||||
|
:Very long argument:
|
||||||
|
Is very specific.
|
||||||
|
|
||||||
|
To demonstrate usage - useful mainly when the syntax is complicated, list the
|
||||||
|
full command with arguments in monospaced font, then indent the next line and
|
||||||
|
describe the effect::
|
||||||
|
|
||||||
|
``resume all``
|
||||||
|
Resumes all suspended constructions.
|
||||||
|
|
||||||
|
If it would be helpful to mention another DFHack command, don't just type the
|
||||||
|
name - add a hyperlink! Specify the link target in backticks, and it will be
|
||||||
|
replaced with the corresponding title and linked: eg ```autolabor```
|
||||||
|
=> `autolabor`. Link targets should be equivalent to the command
|
||||||
|
described (without file extension), and placed above the heading of that
|
||||||
|
section like this::
|
||||||
|
|
||||||
|
.. _autolabor:
|
||||||
|
|
||||||
|
autolabor
|
||||||
|
=========
|
||||||
|
|
||||||
|
Add link targets if you need them, but otherwise plain headings are preferred.
|
||||||
|
Scripts have link targets created automatically.
|
||||||
|
|
||||||
|
|
||||||
|
Building the documentation
|
||||||
|
==========================
|
||||||
|
|
||||||
|
Required dependencies
|
||||||
|
---------------------
|
||||||
|
In order to build the documentation, you must have Python with Sphinx
|
||||||
|
version |sphinx_min_version| or later. Both Python 2.x and 3.x are supported.
|
||||||
|
|
||||||
|
When installing Sphinx from OS package managers, be aware that there is
|
||||||
|
another program called Sphinx, completely unrelated to documentation management.
|
||||||
|
Be sure you are installing the right Sphinx; it may be called ``python-sphinx``,
|
||||||
|
for example. To avoid doubt, ``pip`` can be used instead as detailed below.
|
||||||
|
|
||||||
|
Once you have installed Sphinx, ``sphinx-build --version`` should report the
|
||||||
|
version of Sphinx that you have installed. If this works, CMake should also be
|
||||||
|
able to find Sphinx.
|
||||||
|
|
||||||
|
For more detailed platform-specific instructions, see the sections below:
|
||||||
|
|
||||||
|
.. contents::
|
||||||
|
:local:
|
||||||
|
:backlinks: none
|
||||||
|
|
||||||
|
|
||||||
|
Linux
|
||||||
|
~~~~~
|
||||||
|
Most Linux distributions will include Python by default. If not, start by
|
||||||
|
installing Python (preferably Python 3). On Debian-based distros::
|
||||||
|
|
||||||
|
sudo apt install python3
|
||||||
|
|
||||||
|
Check your package manager to see if Sphinx |sphinx_min_version| or later is
|
||||||
|
available. On Debian-based distros, this package is named ``python3-sphinx``.
|
||||||
|
If this package is new enough, you can install it directly. If not, or if you
|
||||||
|
want to use a newer Sphinx version (which may result in faster builds), you
|
||||||
|
can install Sphinx through the ``pip`` package manager instead. On Debian-based
|
||||||
|
distros, you can install pip with::
|
||||||
|
|
||||||
|
sudo apt install python3-pip
|
||||||
|
|
||||||
|
Once pip is available, you can then install Sphinx with::
|
||||||
|
|
||||||
|
pip3 install sphinx
|
||||||
|
|
||||||
|
If you run this as an unpriviliged user, it may install a local copy of Sphinx
|
||||||
|
for your user only. The ``sphinx-build`` executable will typically end up in
|
||||||
|
``~/.local/bin/`` in this case. Alternatively, you can install Sphinx
|
||||||
|
system-wide by running pip with ``sudo``. In any case, you will need the folder
|
||||||
|
containing ``sphinx-build`` to be in your ``$PATH``.
|
||||||
|
|
||||||
|
macOS
|
||||||
|
~~~~~
|
||||||
|
macOS has Python 2.7 installed by default, but it does not have the pip package manager.
|
||||||
|
|
||||||
|
You can install Homebrew's Python 3, which includes pip, and then install the
|
||||||
|
latest Sphinx using pip::
|
||||||
|
|
||||||
|
brew install python3
|
||||||
|
pip3 install sphinx
|
||||||
|
|
||||||
|
Alternatively, you can simply install Sphinx directly from Homebrew::
|
||||||
|
|
||||||
|
brew install sphinx-doc
|
||||||
|
|
||||||
|
This will install Sphinx for macOS's system Python 2.7, without needing pip.
|
||||||
|
|
||||||
|
Either method works; if you plan to use Python for other purposes, it might best
|
||||||
|
to install Homebrew's Python 3 so that you have the latest Python as well as pip.
|
||||||
|
If not, just installing sphinx-doc for macOS's system Python 2.7 is fine.
|
||||||
|
|
||||||
|
|
||||||
|
Windows
|
||||||
|
~~~~~~~
|
||||||
|
Python for Windows can be downloaded `from python.org <https://www.python.org/downloads/>`_.
|
||||||
|
The latest version of Python 3 is recommended, as it includes pip already.
|
||||||
|
|
||||||
|
You can also install Python and pip through the Chocolatey package manager.
|
||||||
|
After installing Chocolatey as outlined in the `Windows compilation instructions <compile-windows>`,
|
||||||
|
run the following command from an elevated (admin) command prompt (e.g. ``cmd.exe``)::
|
||||||
|
|
||||||
|
choco install python pip -y
|
||||||
|
|
||||||
|
Once you have pip available, you can install Sphinx with the following command::
|
||||||
|
|
||||||
|
pip install sphinx
|
||||||
|
|
||||||
|
Note that this may require opening a new (admin) command prompt if you just
|
||||||
|
installed pip from the same command prompt.
|
||||||
|
|
||||||
|
|
||||||
|
Using CMake
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Enabling the ``BUILD_DOCS`` CMake option will cause the documentation to be built
|
||||||
|
whenever it changes as part of the normal DFHack build process. There are several
|
||||||
|
ways to do this:
|
||||||
|
|
||||||
|
* When initially running CMake, add ``-DBUILD_DOCS:bool=ON`` to your ``cmake``
|
||||||
|
command. For example::
|
||||||
|
|
||||||
|
cmake .. -DCMAKE_BUILD_TYPE:string=Release -DBUILD_DOCS:bool=ON -DCMAKE_INSTALL_PREFIX=<path to DF>
|
||||||
|
|
||||||
|
* If you have already run CMake, you can simply run it again from your build
|
||||||
|
folder to update your configuration::
|
||||||
|
|
||||||
|
cmake .. -DBUILD_DOCS:bool=ON
|
||||||
|
|
||||||
|
* You can edit the ``BUILD_DOCS`` setting in CMakeCache.txt directly
|
||||||
|
|
||||||
|
* You can use the CMake GUI to change ``BUILD_DOCS``
|
||||||
|
|
||||||
|
* On Windows, if you prefer to use the batch scripts, you can run
|
||||||
|
``generate-msvc-gui.bat`` and set ``BUILD_DOCS`` through the GUI. If you are
|
||||||
|
running another file, such as ``generate-msvc-all.bat``, you will need to edit
|
||||||
|
it to add the flag. You can also run ``cmake`` on the command line, similar to
|
||||||
|
other platforms.
|
||||||
|
|
||||||
|
Running Sphinx manually
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
You can also build the documentation without going through CMake, which may be
|
||||||
|
faster. There is a ``docs/build.sh`` script available for Linux and macOS that
|
||||||
|
will run essentially the same command that CMake runs - see the script for
|
||||||
|
options.
|
||||||
|
|
||||||
|
To build the documentation with default options, run the following command from
|
||||||
|
the root DFHack folder::
|
||||||
|
|
||||||
|
sphinx-build . docs/html
|
||||||
|
|
||||||
|
Sphinx has many options to enable clean builds, parallel builds, logging, and
|
||||||
|
more - run ``sphinx-build --help`` for details.
|
||||||
|
|
||||||
|
|
||||||
|
.. _build-changelog:
|
||||||
|
|
||||||
|
Building the changelogs
|
||||||
|
=======================
|
||||||
|
If you have Python installed, but do not want to build all of the documentation,
|
||||||
|
you can build the changelogs with the ``docs/gen_changelog.py`` script. This
|
||||||
|
script provides additional options, including one to build individual changelogs
|
||||||
|
for all DFHack versions - run ``python docs/gen_changelog.py --help`` for details.
|
||||||
|
|
||||||
|
Changelog entries are obtained from ``changelog.txt`` files in multiple repos.
|
||||||
|
This allows changes to be listed in the same repo where they were made. These
|
||||||
|
changelogs are combined as part of the changelog build process:
|
||||||
|
|
||||||
|
* ``docs/changelog.txt`` for changes in the main ``dfhack`` repo
|
||||||
|
* ``scripts/changelog.txt`` for changes made to scripts in the ``scripts`` repo
|
||||||
|
* ``library/xml/changelog.txt`` for changes made in the ``df-structures`` repo
|
||||||
|
|
||||||
|
Building the changelogs generates two files: ``docs/_auto/news.rst`` and
|
||||||
|
``docs/_auto/news-dev.rst``. These correspond to `changelog` and `dev-changelog`
|
||||||
|
and contain changes organized by stable and development DFHack releases,
|
||||||
|
respectively. For example, an entry listed under "0.44.05-alpha1" in
|
||||||
|
changelog.txt will be listed under that version in the development changelog as
|
||||||
|
well, but under "0.44.05-r1" in the stable changelog (assuming that is the
|
||||||
|
closest stable release after 0.44.05-alpha1). An entry listed under a stable
|
||||||
|
release like "0.44.05-r1" in changelog.txt will be listed under that release in
|
||||||
|
both the stable changelog and the development changelog.
|
||||||
|
|
||||||
|
|
||||||
|
Changelog syntax
|
||||||
|
----------------
|
||||||
|
|
||||||
|
.. include:: /docs/changelog.txt
|
||||||
|
:start-after: ===help
|
||||||
|
:end-before: ===end
|
||||||
@ -0,0 +1,149 @@
|
|||||||
|
.. _memory-research:
|
||||||
|
|
||||||
|
###############
|
||||||
|
Memory research
|
||||||
|
###############
|
||||||
|
|
||||||
|
There are a variety of tools that can be used to analyze DF memory - some are
|
||||||
|
listed here. Note that some of these may be old and unmaintained. If you aren't
|
||||||
|
sure what tool would be best for your purposes, feel free to ask for advice (on
|
||||||
|
IRC, Bay12, etc.).
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
|
||||||
|
Cross-platform tools
|
||||||
|
====================
|
||||||
|
|
||||||
|
Ghidra
|
||||||
|
------
|
||||||
|
|
||||||
|
Ghidra is a cross-platform reverse-engineering framework (written in Java)
|
||||||
|
available at https://ghidra-sre.org. It supports analyzing both 32-bit and
|
||||||
|
64-bit executables for all supported DF platforms. There are some custom DFHack
|
||||||
|
Ghidra scripts available in the `df_misc`_ repo (look for ``.java`` files).
|
||||||
|
|
||||||
|
|
||||||
|
IDA Freeware 7.0
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Available from `Hex-Rays <https://www.hex-rays.com/products/ida/support/download_freeware/>`_.
|
||||||
|
Supports analyzing both 32-bit and 64-bit executables for all supported DF platforms.
|
||||||
|
Some ``.idc`` scripts for IDA are available in the `df_misc`_ repo.
|
||||||
|
|
||||||
|
.. _df_misc: https://github.com/DFHack/df_misc
|
||||||
|
|
||||||
|
|
||||||
|
Hopper
|
||||||
|
------
|
||||||
|
|
||||||
|
Runs on macOS and some Linux distributions; available from https://www.hopperapp.com/.
|
||||||
|
`TWBT <https://github.com/mifki/df-twbt/blob/master/PATCHES.md>`_ uses this to produce some patches.
|
||||||
|
|
||||||
|
|
||||||
|
DFHack tools
|
||||||
|
------------
|
||||||
|
|
||||||
|
Plugins
|
||||||
|
~~~~~~~
|
||||||
|
|
||||||
|
There are a few development plugins useful for low-level memory research. They
|
||||||
|
are not built by default, but can be built by setting the ``BUILD_DEVEL``
|
||||||
|
`CMake option <compile-build-options>`. These include:
|
||||||
|
|
||||||
|
- ``check-structures-sanity``, which performs sanity checks on the given DF
|
||||||
|
object. Note that this will crash in several cases, some intentional, so using
|
||||||
|
this with `GDB <linux-gdb>` is recommended.
|
||||||
|
- ``memview``, which produces a hex dump of a given memory range. It also
|
||||||
|
highlights valid pointers, and can be configured to work with `sizecheck`
|
||||||
|
to auto-detect object sizes.
|
||||||
|
- ``vectors``, which can identify instances of ``std::vector`` in a given memory range.
|
||||||
|
|
||||||
|
Scripts
|
||||||
|
~~~~~~~
|
||||||
|
|
||||||
|
Several `development scripts <scripts-devel>` can be useful for memory research.
|
||||||
|
These include (but are not limited to):
|
||||||
|
|
||||||
|
- `devel/dump-offsets`
|
||||||
|
- `devel/find-offsets`
|
||||||
|
- `devel/lsmem`
|
||||||
|
- `devel/sc` (requires `sizecheck`)
|
||||||
|
- `devel/visualize-structure`
|
||||||
|
- Generally, any script starting with ``devel/find``
|
||||||
|
|
||||||
|
.. _sizecheck:
|
||||||
|
|
||||||
|
Sizecheck
|
||||||
|
~~~~~~~~~
|
||||||
|
|
||||||
|
Sizecheck is a custom tool that hooks into the memory allocator and inserts a
|
||||||
|
header indicating the size of every object. The corresponding logic to check for
|
||||||
|
this header when freeing memory usually works, but is inherently not foolproof.
|
||||||
|
You should not count on DF being stable when using this.
|
||||||
|
|
||||||
|
DFHack's implementation of sizecheck is currently only tested on Linux, although
|
||||||
|
it probably also works on macOS. It can be built with the ``BUILD_SIZECHECK``
|
||||||
|
`CMake option <compile-build-options>`, which produces a ``libsizecheck``
|
||||||
|
library installed in the ``hack`` folder. You will need to preload this library
|
||||||
|
manually, by setting ``PRELOAD_LIB`` on Linux (or ``LD_PRELOAD`` if editing
|
||||||
|
the ``dfhack`` launcher script directly), or by editing the ``dfhack``
|
||||||
|
launcher script and adding the library to ``DYLD_INSERT_LIBRARIES`` on macOS.
|
||||||
|
|
||||||
|
There is also an older sizecheck implementation by Mifki available on
|
||||||
|
`GitHub <https://github.com/mifki/df-sizecheck>`__ (``b.cpp`` is the main
|
||||||
|
sizecheck library, and ``win_patch.cpp`` is used for Windows support). To use
|
||||||
|
this with other DFHack tools, you will likely need to edit the header's
|
||||||
|
magic number to match what is used in `devel/sc` (search for a hexadecimal
|
||||||
|
constant starting with ``0x``).
|
||||||
|
|
||||||
|
Legacy tools
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Some very old DFHack tools are available in the `legacy branch on GitHub <https://github.com/dfhack/dfhack/tree/legacy/tools>`_.
|
||||||
|
No attempt is made to support these.
|
||||||
|
|
||||||
|
|
||||||
|
Linux-specific tools
|
||||||
|
====================
|
||||||
|
|
||||||
|
.. _linux-gdb:
|
||||||
|
|
||||||
|
GDB
|
||||||
|
---
|
||||||
|
|
||||||
|
`GDB <https://www.gnu.org/software/gdb/>`_ is technically cross-platform, but
|
||||||
|
tends to work best on Linux, and DFHack currently only offers support for using
|
||||||
|
GDB on 64-bit Linux. To start with GDB, pass ``-g`` to the DFHack launcher
|
||||||
|
script::
|
||||||
|
|
||||||
|
./dfhack -g
|
||||||
|
|
||||||
|
Some basic GDB commands:
|
||||||
|
|
||||||
|
- ``run``: starts DF from the GDB prompt. Any arguments will be passed as
|
||||||
|
command-line arguments to DF (e.g. `load-save` may be useful).
|
||||||
|
- ``bt`` will produce a backtrace if DF crashes.
|
||||||
|
|
||||||
|
See the `official GDB documentation <https://www.gnu.org/software/gdb/documentation/>`_
|
||||||
|
for more details.
|
||||||
|
|
||||||
|
|
||||||
|
df-structures GUI
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
This is a tool written by Angavrilov and available on `GitHub <https://github.com/angavrilov/cl-linux-debug>`__.
|
||||||
|
It only supports 32-bit DF. Some assistance may be available on IRC.
|
||||||
|
|
||||||
|
|
||||||
|
EDB (Evan's debugger)
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Available on `GitHub <https://github.com/eteran/edb-debugger>`__.
|
||||||
|
|
||||||
|
|
||||||
|
Windows-specific tools
|
||||||
|
======================
|
||||||
|
|
||||||
|
Some people have used `Cheat Engine <https://www.cheatengine.org/>`__ for research in the past.
|
||||||
@ -0,0 +1,17 @@
|
|||||||
|
#############
|
||||||
|
Removed tools
|
||||||
|
#############
|
||||||
|
|
||||||
|
This page lists tools (plugins or scripts) that were previously included in
|
||||||
|
DFHack but have been removed. It exists primarily so that internal links still
|
||||||
|
work (e.g. links from the `changelog`).
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
:depth: 1
|
||||||
|
|
||||||
|
.. _warn-stuck-trees:
|
||||||
|
|
||||||
|
warn-stuck-trees
|
||||||
|
================
|
||||||
|
The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01.
|
||||||
@ -1,14 +0,0 @@
|
|||||||
###############
|
|
||||||
Removed scripts
|
|
||||||
###############
|
|
||||||
|
|
||||||
The following scripts were removed for various reasons.
|
|
||||||
|
|
||||||
.. contents::
|
|
||||||
:depth: 2
|
|
||||||
|
|
||||||
.. _warn-stuck-trees:
|
|
||||||
|
|
||||||
warn-stuck-trees
|
|
||||||
================
|
|
||||||
The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01.
|
|
||||||
@ -0,0 +1,34 @@
|
|||||||
|
===================================
|
||||||
|
DF data definitions (DF-structures)
|
||||||
|
===================================
|
||||||
|
|
||||||
|
DFHack's information about DF's data structures is stored in XML files in the
|
||||||
|
`df-structures repository <https://github.com/DFHack/df-structures>`_. If you
|
||||||
|
have `obtained a local copy of the DFHack source <compile-how-to-get-the-code>`,
|
||||||
|
DF-structures is included as a submodule in ``library/xml``.
|
||||||
|
|
||||||
|
Data structure layouts are described in files named with the ``df.*.xml``
|
||||||
|
pattern. This information is transformed by a Perl script (``codegen.pl``) into
|
||||||
|
C++ headers, as well as metadata for the Lua wrapper. This ultimately allows
|
||||||
|
DFHack code to access DF data directly, with the same speed and capabilities as
|
||||||
|
DF itself, which is an advantage over the older out-of-process approach (used
|
||||||
|
by debuggers and utilities like Dwarf Therapist). The main disadvantage of this
|
||||||
|
approach is that any compiled code relying on these layouts will break when DF's
|
||||||
|
layout changes, and will need to be recompiled for every new DF version.
|
||||||
|
|
||||||
|
Addresses of DF global objects and vtables are stored in a separate file,
|
||||||
|
:file:`symbols.xml`. Since these are only absolute addresses, they do not need
|
||||||
|
to be compiled in to DFHack code, and are instead loaded at runtime. This makes
|
||||||
|
fixes and additions to global addresses possible without recompiling DFHack.
|
||||||
|
In an installed copy of DFHack, this file can be found at the root of the
|
||||||
|
``hack`` folder.
|
||||||
|
|
||||||
|
The following pages contain more detailed information about various aspects
|
||||||
|
of DF-structures:
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:titlesonly:
|
||||||
|
|
||||||
|
/library/xml/SYNTAX
|
||||||
|
/library/xml/how-to-update
|
||||||
|
|
||||||
@ -1,285 +1,5 @@
|
|||||||
import collections
|
import os, sys
|
||||||
import copy
|
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'sphinx_extensions'))
|
||||||
import itertools
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
CHANGELOG_PATHS = (
|
from dfhack.changelog import cli_entrypoint
|
||||||
'docs/changelog.txt',
|
cli_entrypoint()
|
||||||
'scripts/changelog.txt',
|
|
||||||
'library/xml/changelog.txt',
|
|
||||||
)
|
|
||||||
|
|
||||||
CHANGELOG_SECTIONS = [
|
|
||||||
'New Plugins',
|
|
||||||
'New Scripts',
|
|
||||||
'New Tweaks',
|
|
||||||
'New Features',
|
|
||||||
'New Internal Commands',
|
|
||||||
'Fixes',
|
|
||||||
'Misc Improvements',
|
|
||||||
'Removed',
|
|
||||||
'API',
|
|
||||||
'Internals',
|
|
||||||
'Lua',
|
|
||||||
'Ruby',
|
|
||||||
'Structures',
|
|
||||||
]
|
|
||||||
|
|
||||||
REPLACEMENTS = {
|
|
||||||
'`search`': '`search-plugin`',
|
|
||||||
}
|
|
||||||
|
|
||||||
def to_title_case(word):
|
|
||||||
if word == word.upper():
|
|
||||||
# Preserve acronyms
|
|
||||||
return word
|
|
||||||
return word[0].upper() + word[1:].lower()
|
|
||||||
|
|
||||||
def find_all_indices(string, substr):
|
|
||||||
start = 0
|
|
||||||
while True:
|
|
||||||
i = string.find(substr, start)
|
|
||||||
if i == -1:
|
|
||||||
return
|
|
||||||
yield i
|
|
||||||
start = i + 1
|
|
||||||
|
|
||||||
def replace_text(string, replacements):
|
|
||||||
for old_text, new_text in replacements.items():
|
|
||||||
new_string = ''
|
|
||||||
new_string_end = 0 # number of characters from string in new_string
|
|
||||||
for i in find_all_indices(string, old_text):
|
|
||||||
if i > 0 and string[i - 1] == '!':
|
|
||||||
# exempt if preceded by '!'
|
|
||||||
new_string += string[new_string_end:i - 1]
|
|
||||||
new_string += old_text
|
|
||||||
else:
|
|
||||||
# copy until this occurrence
|
|
||||||
new_string += string[new_string_end:i]
|
|
||||||
new_string += new_text
|
|
||||||
new_string_end = i + len(old_text)
|
|
||||||
new_string += string[new_string_end:]
|
|
||||||
string = new_string
|
|
||||||
return string
|
|
||||||
|
|
||||||
class ChangelogEntry(object):
|
|
||||||
def __init__(self, text, section, stable_version, dev_version):
|
|
||||||
text = text.lstrip('- ')
|
|
||||||
# normalize section to title case
|
|
||||||
self.section = ' '.join(map(to_title_case, section.strip().split()))
|
|
||||||
self.stable_version = stable_version
|
|
||||||
self.dev_version = dev_version
|
|
||||||
self.dev_only = text.startswith('@')
|
|
||||||
text = text.lstrip('@ ')
|
|
||||||
self.children = []
|
|
||||||
|
|
||||||
split_index = text.find(': ')
|
|
||||||
if split_index != -1:
|
|
||||||
self.feature, description = text[:split_index], text[split_index+1:]
|
|
||||||
if description.strip():
|
|
||||||
self.children.insert(0, description.strip())
|
|
||||||
else:
|
|
||||||
self.feature = text
|
|
||||||
self.feature = self.feature.replace(':\\', ':').rstrip(':')
|
|
||||||
|
|
||||||
self.sort_key = self.feature.upper()
|
|
||||||
|
|
||||||
def __repr__(self):
|
|
||||||
return 'ChangelogEntry(%r, %r)' % (self.feature, self.children)
|
|
||||||
|
|
||||||
def parse_changelog():
|
|
||||||
entries = []
|
|
||||||
|
|
||||||
for fpath in CHANGELOG_PATHS:
|
|
||||||
if not os.path.isfile(fpath):
|
|
||||||
continue
|
|
||||||
with open(fpath) as f:
|
|
||||||
cur_stable = None
|
|
||||||
cur_dev = None
|
|
||||||
cur_section = None
|
|
||||||
last_entry = None
|
|
||||||
multiline = ''
|
|
||||||
for line_id, line in enumerate(f.readlines()):
|
|
||||||
line_id += 1
|
|
||||||
|
|
||||||
if multiline:
|
|
||||||
multiline += line
|
|
||||||
elif '[[[' in line:
|
|
||||||
multiline = line.replace('[[[', '')
|
|
||||||
|
|
||||||
if ']]]' in multiline:
|
|
||||||
line = multiline.replace(']]]', '')
|
|
||||||
multiline = ''
|
|
||||||
elif multiline:
|
|
||||||
continue
|
|
||||||
|
|
||||||
if not line.strip() or line.startswith('==='):
|
|
||||||
continue
|
|
||||||
|
|
||||||
if line.startswith('##'):
|
|
||||||
cur_section = line.lstrip('#').strip()
|
|
||||||
elif line.startswith('#'):
|
|
||||||
cur_dev = line.lstrip('#').strip().lower()
|
|
||||||
if ('alpha' not in cur_dev and 'beta' not in cur_dev and
|
|
||||||
'rc' not in cur_dev):
|
|
||||||
cur_stable = cur_dev
|
|
||||||
elif line.startswith('-'):
|
|
||||||
if not cur_stable or not cur_dev or not cur_section:
|
|
||||||
raise ValueError(
|
|
||||||
'%s:%i: Entry without section' % (fpath, line_id))
|
|
||||||
last_entry = ChangelogEntry(line.strip(), cur_section,
|
|
||||||
cur_stable, cur_dev)
|
|
||||||
entries.append(last_entry)
|
|
||||||
elif line.lstrip().startswith('-'):
|
|
||||||
if not cur_stable or not cur_dev:
|
|
||||||
raise ValueError(
|
|
||||||
'%s:%i: Sub-entry without section' % (fpath, line_id))
|
|
||||||
if not last_entry:
|
|
||||||
raise ValueError(
|
|
||||||
'%s:%i: Sub-entry without parent' % (fpath, line_id))
|
|
||||||
last_entry.children.append(line.strip('- \n'))
|
|
||||||
else:
|
|
||||||
raise ValueError('Invalid line: ' + line)
|
|
||||||
|
|
||||||
if not entries:
|
|
||||||
raise RuntimeError('No changelog files with contents found')
|
|
||||||
|
|
||||||
return entries
|
|
||||||
|
|
||||||
def consolidate_changelog(all_entries):
|
|
||||||
for sections in all_entries.values():
|
|
||||||
for section, entries in sections.items():
|
|
||||||
# sort() is stable, so reverse entries so that older entries for the
|
|
||||||
# same feature are on top
|
|
||||||
entries.reverse()
|
|
||||||
entries.sort(key=lambda entry: entry.sort_key)
|
|
||||||
new_entries = []
|
|
||||||
for feature, group in itertools.groupby(entries,
|
|
||||||
lambda e: e.feature):
|
|
||||||
old_entries = list(group)
|
|
||||||
children = list(itertools.chain(*[entry.children
|
|
||||||
for entry in old_entries]))
|
|
||||||
new_entry = copy.deepcopy(old_entries[0])
|
|
||||||
new_entry.children = children
|
|
||||||
new_entries.append(new_entry)
|
|
||||||
entries[:] = new_entries
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def print_changelog(versions, all_entries, path, replace=True, prefix=''):
|
|
||||||
# all_entries: version -> section -> entry
|
|
||||||
with open(path, 'w') as f:
|
|
||||||
def write(line):
|
|
||||||
if replace:
|
|
||||||
line = replace_text(line, REPLACEMENTS)
|
|
||||||
f.write(prefix + line + '\n')
|
|
||||||
for version in versions:
|
|
||||||
sections = all_entries[version]
|
|
||||||
if not sections:
|
|
||||||
continue
|
|
||||||
version = 'DFHack ' + version
|
|
||||||
write(version)
|
|
||||||
write('=' * len(version))
|
|
||||||
write('')
|
|
||||||
for section in CHANGELOG_SECTIONS:
|
|
||||||
entries = sections[section]
|
|
||||||
if not entries:
|
|
||||||
continue
|
|
||||||
write(section)
|
|
||||||
write('-' * len(section))
|
|
||||||
for entry in entries:
|
|
||||||
if len(entry.children) == 1:
|
|
||||||
write('- ' + entry.feature + ': ' +
|
|
||||||
entry.children[0].strip('- '))
|
|
||||||
continue
|
|
||||||
elif entry.children:
|
|
||||||
write('- ' + entry.feature + ':')
|
|
||||||
write('')
|
|
||||||
for child in entry.children:
|
|
||||||
write(' - ' + child)
|
|
||||||
write('')
|
|
||||||
else:
|
|
||||||
write('- ' + entry.feature)
|
|
||||||
write('')
|
|
||||||
write('')
|
|
||||||
|
|
||||||
|
|
||||||
def generate_changelog(all=False):
|
|
||||||
entries = parse_changelog()
|
|
||||||
|
|
||||||
# scan for unrecognized sections
|
|
||||||
for entry in entries:
|
|
||||||
if entry.section not in CHANGELOG_SECTIONS:
|
|
||||||
raise RuntimeWarning('Unknown section: ' + entry.section)
|
|
||||||
|
|
||||||
# ordered versions
|
|
||||||
versions = ['future']
|
|
||||||
# map versions to stable versions
|
|
||||||
stable_version_map = {}
|
|
||||||
# version -> section -> entry
|
|
||||||
stable_entries = collections.defaultdict(lambda:
|
|
||||||
collections.defaultdict(list))
|
|
||||||
dev_entries = collections.defaultdict(lambda:
|
|
||||||
collections.defaultdict(list))
|
|
||||||
|
|
||||||
for entry in entries:
|
|
||||||
# build list of all versions
|
|
||||||
if entry.dev_version not in versions:
|
|
||||||
versions.append(entry.dev_version)
|
|
||||||
stable_version_map.setdefault(entry.dev_version, entry.stable_version)
|
|
||||||
|
|
||||||
if not entry.dev_only:
|
|
||||||
# add non-dev-only entries to both changelogs
|
|
||||||
stable_entries[entry.stable_version][entry.section].append(entry)
|
|
||||||
dev_entries[entry.dev_version][entry.section].append(entry)
|
|
||||||
|
|
||||||
consolidate_changelog(stable_entries)
|
|
||||||
consolidate_changelog(dev_entries)
|
|
||||||
|
|
||||||
print_changelog(versions, stable_entries, 'docs/_auto/news.rst')
|
|
||||||
print_changelog(versions, dev_entries, 'docs/_auto/news-dev.rst')
|
|
||||||
|
|
||||||
if all:
|
|
||||||
for version in versions:
|
|
||||||
if version not in stable_version_map:
|
|
||||||
print('warn: skipping ' + version)
|
|
||||||
continue
|
|
||||||
if stable_version_map[version] == version:
|
|
||||||
version_entries = {version: stable_entries[version]}
|
|
||||||
else:
|
|
||||||
version_entries = {version: dev_entries[version]}
|
|
||||||
print_changelog([version], version_entries,
|
|
||||||
'docs/_changelogs/%s-github.txt' % version,
|
|
||||||
replace=False)
|
|
||||||
print_changelog([version], version_entries,
|
|
||||||
'docs/_changelogs/%s-reddit.txt' % version,
|
|
||||||
replace=False,
|
|
||||||
prefix='> ')
|
|
||||||
|
|
||||||
return entries
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
import argparse
|
|
||||||
parser = argparse.ArgumentParser()
|
|
||||||
parser.add_argument('-a', '--all', action='store_true',
|
|
||||||
help='Print changelogs for all versions to docs/_changelogs')
|
|
||||||
parser.add_argument('-c', '--check', action='store_true',
|
|
||||||
help='Check that all entries are printed')
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
os.chdir(os.path.abspath(os.path.dirname(__file__)))
|
|
||||||
os.chdir('..')
|
|
||||||
entries = generate_changelog(all=args.all)
|
|
||||||
|
|
||||||
if args.check:
|
|
||||||
with open('docs/_auto/news.rst') as f:
|
|
||||||
content_stable = f.read()
|
|
||||||
with open('docs/_auto/news-dev.rst') as f:
|
|
||||||
content_dev = f.read()
|
|
||||||
for entry in entries:
|
|
||||||
for description in entry.children:
|
|
||||||
if not entry.dev_only and description not in content_stable:
|
|
||||||
print('stable missing: ' + description)
|
|
||||||
if description not in content_dev:
|
|
||||||
print('dev missing: ' + description)
|
|
||||||
|
|||||||
@ -0,0 +1,13 @@
|
|||||||
|
============
|
||||||
|
About DFHack
|
||||||
|
============
|
||||||
|
|
||||||
|
These pages contain information about the general DFHack project.
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 1
|
||||||
|
|
||||||
|
/docs/NEWS
|
||||||
|
/docs/Authors
|
||||||
|
/LICENSE
|
||||||
|
/docs/Removed
|
||||||
@ -0,0 +1,19 @@
|
|||||||
|
========================
|
||||||
|
DFHack Development Guide
|
||||||
|
========================
|
||||||
|
|
||||||
|
These are pages relevant to people developing for DFHack.
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 1
|
||||||
|
|
||||||
|
/docs/Dev-intro
|
||||||
|
/docs/Contributing
|
||||||
|
/docs/Compile
|
||||||
|
/docs/NEWS-dev
|
||||||
|
/docs/Lua API
|
||||||
|
/docs/Documentation
|
||||||
|
/docs/Structures-intro
|
||||||
|
/docs/Memory-research
|
||||||
|
/docs/Binpatches
|
||||||
|
|
||||||
@ -0,0 +1,307 @@
|
|||||||
|
import collections
|
||||||
|
import copy
|
||||||
|
import itertools
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from sphinx.errors import ExtensionError, SphinxError, SphinxWarning
|
||||||
|
|
||||||
|
from dfhack.util import DFHACK_ROOT, DOCS_ROOT
|
||||||
|
|
||||||
|
CHANGELOG_PATHS = (
|
||||||
|
'docs/changelog.txt',
|
||||||
|
'scripts/changelog.txt',
|
||||||
|
'library/xml/changelog.txt',
|
||||||
|
)
|
||||||
|
CHANGELOG_PATHS = (os.path.join(DFHACK_ROOT, p) for p in CHANGELOG_PATHS)
|
||||||
|
|
||||||
|
CHANGELOG_SECTIONS = [
|
||||||
|
'New Plugins',
|
||||||
|
'New Scripts',
|
||||||
|
'New Tweaks',
|
||||||
|
'New Features',
|
||||||
|
'New Internal Commands',
|
||||||
|
'Fixes',
|
||||||
|
'Misc Improvements',
|
||||||
|
'Removed',
|
||||||
|
'API',
|
||||||
|
'Internals',
|
||||||
|
'Lua',
|
||||||
|
'Ruby',
|
||||||
|
'Structures',
|
||||||
|
]
|
||||||
|
|
||||||
|
REPLACEMENTS = {
|
||||||
|
'`search`': '`search-plugin`',
|
||||||
|
}
|
||||||
|
|
||||||
|
def to_title_case(word):
|
||||||
|
if word == word.upper():
|
||||||
|
# Preserve acronyms
|
||||||
|
return word
|
||||||
|
return word[0].upper() + word[1:].lower()
|
||||||
|
|
||||||
|
def find_all_indices(string, substr):
|
||||||
|
start = 0
|
||||||
|
while True:
|
||||||
|
i = string.find(substr, start)
|
||||||
|
if i == -1:
|
||||||
|
return
|
||||||
|
yield i
|
||||||
|
start = i + 1
|
||||||
|
|
||||||
|
def replace_text(string, replacements):
|
||||||
|
for old_text, new_text in replacements.items():
|
||||||
|
new_string = ''
|
||||||
|
new_string_end = 0 # number of characters from string in new_string
|
||||||
|
for i in find_all_indices(string, old_text):
|
||||||
|
if i > 0 and string[i - 1] == '!':
|
||||||
|
# exempt if preceded by '!'
|
||||||
|
new_string += string[new_string_end:i - 1]
|
||||||
|
new_string += old_text
|
||||||
|
else:
|
||||||
|
# copy until this occurrence
|
||||||
|
new_string += string[new_string_end:i]
|
||||||
|
new_string += new_text
|
||||||
|
new_string_end = i + len(old_text)
|
||||||
|
new_string += string[new_string_end:]
|
||||||
|
string = new_string
|
||||||
|
return string
|
||||||
|
|
||||||
|
class ChangelogEntry(object):
|
||||||
|
def __init__(self, text, section, stable_version, dev_version):
|
||||||
|
text = text.lstrip('- ')
|
||||||
|
# normalize section to title case
|
||||||
|
self.section = ' '.join(map(to_title_case, section.strip().split()))
|
||||||
|
self.stable_version = stable_version
|
||||||
|
self.dev_version = dev_version
|
||||||
|
self.dev_only = text.startswith('@')
|
||||||
|
text = text.lstrip('@ ')
|
||||||
|
self.children = []
|
||||||
|
|
||||||
|
split_index = text.find(': ')
|
||||||
|
if split_index != -1:
|
||||||
|
self.feature, description = text[:split_index], text[split_index+1:]
|
||||||
|
if description.strip():
|
||||||
|
self.children.insert(0, description.strip())
|
||||||
|
else:
|
||||||
|
self.feature = text
|
||||||
|
self.feature = self.feature.replace(':\\', ':').rstrip(':')
|
||||||
|
|
||||||
|
self.sort_key = self.feature.upper()
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return 'ChangelogEntry(%r, %r)' % (self.feature, self.children)
|
||||||
|
|
||||||
|
def parse_changelog():
|
||||||
|
entries = []
|
||||||
|
|
||||||
|
for fpath in CHANGELOG_PATHS:
|
||||||
|
if not os.path.isfile(fpath):
|
||||||
|
continue
|
||||||
|
with open(fpath) as f:
|
||||||
|
cur_stable = None
|
||||||
|
cur_dev = None
|
||||||
|
cur_section = None
|
||||||
|
last_entry = None
|
||||||
|
multiline = ''
|
||||||
|
for line_id, line in enumerate(f.readlines()):
|
||||||
|
line_id += 1
|
||||||
|
|
||||||
|
if multiline:
|
||||||
|
multiline += line
|
||||||
|
elif '[[[' in line:
|
||||||
|
multiline = line.replace('[[[', '')
|
||||||
|
|
||||||
|
if ']]]' in multiline:
|
||||||
|
line = multiline.replace(']]]', '')
|
||||||
|
multiline = ''
|
||||||
|
elif multiline:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not line.strip() or line.startswith('==='):
|
||||||
|
continue
|
||||||
|
|
||||||
|
if line.startswith('##'):
|
||||||
|
cur_section = line.lstrip('#').strip()
|
||||||
|
elif line.startswith('#'):
|
||||||
|
cur_dev = line.lstrip('#').strip().lower()
|
||||||
|
if ('alpha' not in cur_dev and 'beta' not in cur_dev and
|
||||||
|
'rc' not in cur_dev):
|
||||||
|
cur_stable = cur_dev
|
||||||
|
elif line.startswith('-'):
|
||||||
|
if not cur_stable or not cur_dev or not cur_section:
|
||||||
|
raise ValueError(
|
||||||
|
'%s:%i: Entry without section' % (fpath, line_id))
|
||||||
|
last_entry = ChangelogEntry(line.strip(), cur_section,
|
||||||
|
cur_stable, cur_dev)
|
||||||
|
entries.append(last_entry)
|
||||||
|
elif line.lstrip().startswith('-'):
|
||||||
|
if not cur_stable or not cur_dev:
|
||||||
|
raise ValueError(
|
||||||
|
'%s:%i: Sub-entry without section' % (fpath, line_id))
|
||||||
|
if not last_entry:
|
||||||
|
raise ValueError(
|
||||||
|
'%s:%i: Sub-entry without parent' % (fpath, line_id))
|
||||||
|
last_entry.children.append(line.strip('- \n'))
|
||||||
|
else:
|
||||||
|
raise ValueError('Invalid line: ' + line)
|
||||||
|
|
||||||
|
if not entries:
|
||||||
|
raise RuntimeError('No changelog files with contents found')
|
||||||
|
|
||||||
|
return entries
|
||||||
|
|
||||||
|
def consolidate_changelog(all_entries):
|
||||||
|
for sections in all_entries.values():
|
||||||
|
for section, entries in sections.items():
|
||||||
|
# sort() is stable, so reverse entries so that older entries for the
|
||||||
|
# same feature are on top
|
||||||
|
entries.reverse()
|
||||||
|
entries.sort(key=lambda entry: entry.sort_key)
|
||||||
|
new_entries = []
|
||||||
|
for feature, group in itertools.groupby(entries,
|
||||||
|
lambda e: e.feature):
|
||||||
|
old_entries = list(group)
|
||||||
|
children = list(itertools.chain(*[entry.children
|
||||||
|
for entry in old_entries]))
|
||||||
|
new_entry = copy.deepcopy(old_entries[0])
|
||||||
|
new_entry.children = children
|
||||||
|
new_entries.append(new_entry)
|
||||||
|
entries[:] = new_entries
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def print_changelog(versions, all_entries, path, replace=True, prefix=''):
|
||||||
|
# all_entries: version -> section -> entry
|
||||||
|
with open(path, 'w') as f:
|
||||||
|
def write(line):
|
||||||
|
if replace:
|
||||||
|
line = replace_text(line, REPLACEMENTS)
|
||||||
|
f.write(prefix + line + '\n')
|
||||||
|
for version in versions:
|
||||||
|
sections = all_entries[version]
|
||||||
|
if not sections:
|
||||||
|
continue
|
||||||
|
version = 'DFHack ' + version
|
||||||
|
write(version)
|
||||||
|
write('=' * len(version))
|
||||||
|
write('')
|
||||||
|
for section in CHANGELOG_SECTIONS:
|
||||||
|
entries = sections[section]
|
||||||
|
if not entries:
|
||||||
|
continue
|
||||||
|
write(section)
|
||||||
|
write('-' * len(section))
|
||||||
|
for entry in entries:
|
||||||
|
if len(entry.children) == 1:
|
||||||
|
write('- ' + entry.feature + ': ' +
|
||||||
|
entry.children[0].strip('- '))
|
||||||
|
continue
|
||||||
|
elif entry.children:
|
||||||
|
write('- ' + entry.feature + ':')
|
||||||
|
write('')
|
||||||
|
for child in entry.children:
|
||||||
|
write(' - ' + child)
|
||||||
|
write('')
|
||||||
|
else:
|
||||||
|
write('- ' + entry.feature)
|
||||||
|
write('')
|
||||||
|
write('')
|
||||||
|
|
||||||
|
|
||||||
|
def generate_changelog(all=False):
|
||||||
|
entries = parse_changelog()
|
||||||
|
|
||||||
|
# scan for unrecognized sections
|
||||||
|
for entry in entries:
|
||||||
|
if entry.section not in CHANGELOG_SECTIONS:
|
||||||
|
raise SphinxWarning('Unknown section: ' + entry.section)
|
||||||
|
|
||||||
|
# ordered versions
|
||||||
|
versions = ['future']
|
||||||
|
# map versions to stable versions
|
||||||
|
stable_version_map = {}
|
||||||
|
# version -> section -> entry
|
||||||
|
stable_entries = collections.defaultdict(lambda:
|
||||||
|
collections.defaultdict(list))
|
||||||
|
dev_entries = collections.defaultdict(lambda:
|
||||||
|
collections.defaultdict(list))
|
||||||
|
|
||||||
|
for entry in entries:
|
||||||
|
# build list of all versions
|
||||||
|
if entry.dev_version not in versions:
|
||||||
|
versions.append(entry.dev_version)
|
||||||
|
stable_version_map.setdefault(entry.dev_version, entry.stable_version)
|
||||||
|
|
||||||
|
if not entry.dev_only:
|
||||||
|
# add non-dev-only entries to both changelogs
|
||||||
|
stable_entries[entry.stable_version][entry.section].append(entry)
|
||||||
|
dev_entries[entry.dev_version][entry.section].append(entry)
|
||||||
|
|
||||||
|
consolidate_changelog(stable_entries)
|
||||||
|
consolidate_changelog(dev_entries)
|
||||||
|
|
||||||
|
print_changelog(versions, stable_entries, os.path.join(DOCS_ROOT, '_auto/news.rst'))
|
||||||
|
print_changelog(versions, dev_entries, os.path.join(DOCS_ROOT, '_auto/news-dev.rst'))
|
||||||
|
|
||||||
|
if all:
|
||||||
|
for version in versions:
|
||||||
|
if version not in stable_version_map:
|
||||||
|
print('warn: skipping ' + version)
|
||||||
|
continue
|
||||||
|
if stable_version_map[version] == version:
|
||||||
|
version_entries = {version: stable_entries[version]}
|
||||||
|
else:
|
||||||
|
version_entries = {version: dev_entries[version]}
|
||||||
|
print_changelog([version], version_entries,
|
||||||
|
os.path.join(DOCS_ROOT, '_changelogs/%s-github.txt' % version),
|
||||||
|
replace=False)
|
||||||
|
print_changelog([version], version_entries,
|
||||||
|
os.path.join(DOCS_ROOT, '_changelogs/%s-reddit.txt' % version),
|
||||||
|
replace=False,
|
||||||
|
prefix='> ')
|
||||||
|
|
||||||
|
return entries
|
||||||
|
|
||||||
|
def cli_entrypoint():
|
||||||
|
import argparse
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('-a', '--all', action='store_true',
|
||||||
|
help='Print changelogs for all versions to docs/_changelogs')
|
||||||
|
parser.add_argument('-c', '--check', action='store_true',
|
||||||
|
help='Check that all entries are printed')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
entries = generate_changelog(all=args.all)
|
||||||
|
|
||||||
|
if args.check:
|
||||||
|
with open(os.path.join(DOCS_ROOT, '_auto/news.rst')) as f:
|
||||||
|
content_stable = f.read()
|
||||||
|
with open(os.path.join(DOCS_ROOT, '_auto/news-dev.rst')) as f:
|
||||||
|
content_dev = f.read()
|
||||||
|
for entry in entries:
|
||||||
|
for description in entry.children:
|
||||||
|
if not entry.dev_only and description not in content_stable:
|
||||||
|
print('stable missing: ' + description)
|
||||||
|
if description not in content_dev:
|
||||||
|
print('dev missing: ' + description)
|
||||||
|
|
||||||
|
|
||||||
|
def sphinx_entrypoint(app, config):
|
||||||
|
try:
|
||||||
|
generate_changelog()
|
||||||
|
except SphinxError:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
raise ExtensionError(str(e), e)
|
||||||
|
|
||||||
|
|
||||||
|
def setup(app):
|
||||||
|
app.connect('config-inited', sphinx_entrypoint)
|
||||||
|
|
||||||
|
return {
|
||||||
|
'version': '0.1',
|
||||||
|
'parallel_read_safe': True,
|
||||||
|
'parallel_write_safe': True,
|
||||||
|
}
|
||||||
@ -0,0 +1,7 @@
|
|||||||
|
import os
|
||||||
|
|
||||||
|
DFHACK_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
||||||
|
DOCS_ROOT = os.path.join(DFHACK_ROOT, 'docs')
|
||||||
|
|
||||||
|
if not os.path.isdir(DOCS_ROOT):
|
||||||
|
raise ReferenceError('docs root not found: %s' % DOCS_ROOT)
|
||||||
@ -0,0 +1,6 @@
|
|||||||
|
<p class="logo">
|
||||||
|
<a href="{{ pathto(master_doc) }}">
|
||||||
|
<img class="logo" src="{{ pathto('_static/' ~ theme_logo, 1) }}" alt="Logo"/>
|
||||||
|
<h3 class="logo logo-name">Home</h3>
|
||||||
|
</a>
|
||||||
|
</p>
|
||||||
@ -1 +1 @@
|
|||||||
Subproject commit 0792fc0202fb6a04bfdaa262bc36a3b14c8581e5
|
Subproject commit b9028b0bb9ad40d3ad4dc3f10934bb61aa16629b
|
||||||
@ -1 +1 @@
|
|||||||
Subproject commit b9fc28836f34f7ce1be64de94afd90184a341c7d
|
Subproject commit bdab71c99a0a7cc268ca517a0cd3f0a5fb41042a
|
||||||
@ -1 +1 @@
|
|||||||
Subproject commit 2079b9fb69b8b4db48aa35ec54a96f5cca7cc8ef
|
Subproject commit e8de92efb73d5ef4d0b52df000d60d3350f07a37
|
||||||
Loading…
Reference in New Issue