Pmodules
pmodules allow users to augment snapshots by performing custom analysis and data structure pretty printing in response to specific events.
In this tutorial, we'll write a basic pmodule using the Lua API. The C API is similar; for that, consult the documentation provided in /opt/backtrace/include/ptrace/pmodule.h from the backtrace-ptrace-modules package. For a full reference of the Lua API, see the Pmodule API page on this site. The example module written here is reproduced in full at the top of the Pmodule API page.
From a pmodule's perspective, a snapshot is generated during two phases: attach and postattach. We are in the attach phase when the ptrace tracer is attached to the target process, during which threads, frames and variables are extracted. We are in the postattach phase after the ptrace tracer detaches from the process, at which point we have a fully populated backtrace object.
A pmodule may react to any of these events by registering a callback. For reference, the possible events are:
- variable: A new variable has been extracted (while attached to the process).
- frame: A new frame has been extracted (while attached).
- thread: A new thread has been extracted (while attached).
- preattach: Before ptrace attaches to the process.
- postattach: After ptrace detaches from the process.
- attach: Right after attachment, before extraction.
Each callback will receive as arguments the objects relevant to the event.
- variable: A pmodule variable object and the underlying bt variable object. bt variables will be discussed below.
- frame: A frame object.
- thread: A thread object.
- preattach: A backtrace object.
- postattach: A backtrace object.
- attach: A backtrace object.
As an example of when you'd want a specific callback, you might want to pretty print a specific data structure if it appears in your snapshot, so you'd register a variable event callback with a filter specifying the type of variable you're interested in (we'll cover filters in more detail below).
These event callbacks typically perform some analysis or additional data extraction/formatting, and augment the snapshot via additional classifiers, annotations, and the like.
Now that we have a better understanding of a pmodule's event-driven design, let's start defining our module. At the global (chunk) scope, call pmodule.define like so:
-- The following functions are stubs; we'll define a real load function later.
pmodule.define{
id = "pmodule_lua",
load = function () pmodule.log(pmodule.log_level.warning, "pm_load") end,
unload = function () pmodule.log(pmodule.log_level.warning, "pm_unload") end
}
id and load are required fields; unload is optional. They are used as such:
- id: A string identifying your module.
- load: A function called to register all event callbacks. This is executed before the trace begins.
- unload: A function called before ptrace exit to perform any cleanup required by the module.
pmodule.define will register the module with the pmodule subsystem. After this, the registered loadfunction will be called to establish event handlers as specified. Let's look at an example load:
function pm_load()
pmodule.register(pmodule.event.postattach, postattach_cb)
local m = pmodule.match()
m:add_object("crash")
m:add_file("crash.c")
m:add_frame_symbol("recurse", pmodule.match_type.substr)
m:add_variable_base_type("crash_", pmodule.match_type.substr)
m:add_variable_ptrace_type(pmodule.variable_type.tuple)
pmodule.register(pmodule.event.variable,
function(var)
var:annotate(pmodule.annotation.critical, "lua: struct var")
end, m)
m:reset()
m:set_fault()
pmodule.register(pmodule.event.frame, frame_cb, m)
end
Here, we're interested in three types of events: the postattach event, variable extraction events, and frame events. We'll get into some examples of what we can do in each of them, but for now, let's focus on the match filters, starting with the one used for the variable event callback.
All of the object event callbacks (variable, frame, and thread) may be filtered by a match object. Here, we're saying we're interested in a variable extracted from the object file crash, the compilation unit crash.c, whose frame's symbol contains recurse, whose base type contains crash_, and whose variable type is a tuple (i.e. a struct). Since what we want to do with this variable is pretty simple (annotate it with a useless message), we'll just register the event with an anonymous function and this match filter.
local m = pmodule.match()
m:add_object("crash")
m:add_file("crash.c")
m:add_frame_symbol("recurse", pmodule.match_type.substr)
m:add_variable_base_type("crash_", pmodule.match_type.substr)
m:add_variable_ptrace_type(pmodule.variable_type.tuple)
pmodule.register(pmodule.event.variable,
function(var)
var:annotate(pmodule.annotation.critical, "lua: struct var")
end, m)
Next, let's look at the frame event registration. Here, we're interested in faulting frames, so we reset the match object (you can also just use a different one), set it to match faulted objects, and then register the event using that.
m:reset()
m:set_fault()
pmodule.register(pmodule.event.frame, frame_cb, m)
The postattach event registration call is pretty vanilla, but we'll go deeper into that callback later.
Now that we've registered all our callbacks, our module is ready to start handling events. ptrace will execute its registered callbacks, assuming any specified match filters pass. Let's look at what we want to do when a faulted frame is extracted:
local function frame_cb(fr)
for v, i in fr:fprm() do
v:annotate(pmodule.annotation.critical, "lua: fprm %d", i)
end
local signal = fr:siginfo()
if signal then
fr:backtrace():annotate(
pmodule.annotation.json,
'{"json": {"context": "Signal", \z
"Reason": "%s", \z
"Populated": "%s", \z
"Address": "%x", \z
"Num": "%d", \z
"Code": "%d", \z
"String": "%s"}}',
signal:reason(),
tostring(signal:address_populated()),
signal:address(),
signal:num(),
signal:code(),
tostring(signal))
end
end
Pretty simple. First, we want to annotate all parameters of the faulting frame (this is for illustrative purposes only; hydra will indicate all frame parameters by surrounding them with ( )). For many objects (we'll discuss more later), pmodules support Lua's generic for interface. Here, we can simply iterate over the frame's parameters, which will give us a variable object and a parameter index per iteration.
Note: all four main objects (variables, frames, threads, and the overarching backtrace) may be annotated.
Next, we want to pretty print signal information (again, this is just an example; hydra already has this pretty-printed under the faulting frame). We'll annotate the backtrace object here, so the pretty printed data will be displayed in hydra's Process pane. We'll use the json annotation type for this -- refer to the Pmodule API documentation for more details on the expected format.
Let's look at the postattach callback next:
local function postattach_cb(bt)
local m = pmodule.match()
m:set_fault()
-- Example iterators.
for thr in bt, m do
for fr in thr do
for var in fr do
if var:type() == pmodule.variable_type.reference then
local addr = var:value()
var:annotate(pmodule.annotation.critical,
"[%x] %s - example annotation",
var:value(), var:name())
end
end
end
end
-- Example global variable iteration.
pmodule.log(pmodule.log_level.warning, "Global variables:");
for var, object, cu in bt:variables(), {object = "crash", cu = "invalid_write.c"} do
pmodule.log(pmodule.log_level.warning,
"name: %s, value: %s, object: %s, cu: %s",
var:name(), tostring(var:value()), object, cu)
end
-- Example TLS variable iteration.
for thr in bt do
pmodule.log(pmodule.log_level.warning, "TLS variables:");
for var, object, cu in thr:variables(), {cu = "hang"} do
pmodule.log(pmodule.log_level.warning,
"name: %s, value: %s, object: %s, \z
cu: %s",
var:name(), tostring(var:value()), object, cu)
end
end
-- Example global variable lookup by name.
for var in bt:variables(), {name = "global_version"} do
local str = pmodule.address_read_string(var:value(), 256)
pmodule.log(pmodule.log_level.warning,
"[%x] string: %s", var:value(), str)
end
pmodule.log(pmodule.log_level.warning, "process state: %d",
bt:process_state())
bt:add_kv_int("lua_key1", 42)
bt:add_kv_string("lua_key2", "lua_value")
bt:add_classifier("lua")
end
Here, we start with iteration. The backtrace object, threads, and frames may be iterated over (again using Lua's generic for interface). Each of these iterators accept an optional match object via the invariant state (the second expression in the for's expression list). In the first iterator, we're iterating over all faulting threads.
local m = pmodule.match()
m:set_fault()
-- Example iterators.
for thr in bt, m do
for fr in thr do
for var in fr do
if var:type() == pmodule.variable_type.reference then
local addr = var:value()
var:annotate(pmodule.annotation.critical,
"[%x] %s - example annotation",
var:value(), var:name())
end
end
end
end
Note: the full backtrace will be populated only within the postattach callback (which is why we do the full iteration example in there). In the thread callback, you're guaranteed the full thread will be populated (with its frames and variables). In the frame callback, you're guaranteed the full frame will be populated (with its variables).
Next, we'll cover global and TLS variable iteration.
-- Example global variable iteration.
pmodule.log(pmodule.log_level.warning, "Global variables:");
for var, object, cu in bt:variables(), {object = "crash", cu = "invalid_write.c"} do
pmodule.log(pmodule.log_level.warning,
"name: %s, value: %s, object: %s, cu: %s",
var:name(), tostring(var:value()), object, cu)
end
-- Example TLS variable iteration.
for thr in bt do
pmodule.log(pmodule.log_level.warning, "TLS variables:");
for var, object, cu in thr:variables(), {cu = "hang"} do
pmodule.log(pmodule.log_level.warning,
"name: %s, value: %s, object: %s, \z
cu: %s",
var:name(), tostring(var:value()), object, cu)
end
end
These iterations use the same generic for interface the previously mentioned thread, frame, etc. iterators use, with one difference - the filter (invariant state) here is a table instead of a match object. The supported fields are:
- object: the desired object file (substring match)
- cu: the desired compilation unit (substring match)
- name: the variable's name (exact match)
All of these fields (and the filter table itself) are optional. Each iteration returns a variable object, the object file name, and the cu name.
Let's say we're interested in a particular global variable - some variable containing the version of our program:
-- Example global variable lookup by name.
for var in bt:variables(), {name = "global_version"} do
local str = pmodule.address_read_string(var:value(), 256)
pmodule.log(pmodule.log_level.warning,
"[%x] string: %s", var:value(), str)
end
To find it, we use the global variable iterator with a name filter. This is a C-string variable (i.e. a pointer), so we'll need to use the global read API -- pmodule.address_read_string -- to actually read the string.
Suppose we've noticed something interesting (perhaps application-specific) about the data in our snapshot, and we want to classify the snapshot according to this. Your pmodule may use the following API for that:
-- bt is a backtrace object.
bt:add_kv_int("lua_key1", 42)
bt:add_kv_string("lua_key2", "lua_value")
bt:add_classifier("lua")
Previously, we mentioned bt_variables. Pmodules can use these to access specific fields in structures and iterate over arrays, linked lists, and other data structures. A full example is reproduced below. Note: bt_variables themselves only allow access to member fields, which return new bt_variables; to extract actual data from them, they must first be synthesized into pmodule variables.
--[[
-- Assume we have these definitions:
struct nested {
int c;
};
struct linkedstruct {
STAILQ_ENTRY(linkedstruct) linkage;
int v;
};
struct somestruct {
int a;
double b;
struct nested n;
struct nested *np;
struct nested **npp;
struct nested ***nppp;
STAILQ_HEAD(, linkedstruct) list;
const char *s;
};
struct array_struct {
struct somestruct *kids;
int len;
};
-- And these global variables:
static struct somestruct *some_struct_g;
static struct array_struct papa;
-- And the following population code:
papa.kids = calloc(10, sizeof(struct somestruct));
if (papa.kids == NULL) {
fprintf(stderr, "allocation failure");
exit(EXIT_FAILURE);
}
papa.len = 10;
some_struct_g = calloc(1, sizeof *some_struct_g);
if (some_struct_g == NULL) {
fprintf(stderr, "allocation failure");
exit(EXIT_FAILURE);
}
STAILQ_INIT(&some_struct_g->list);
for (z = 0; z < 10; ++z) {
struct linkedstruct *l;
l = calloc(1, sizeof *l);
if (l == NULL) {
fprintf(stderr, "allocation failure");
exit(EXIT_FAILURE);
}
l->v = z;
STAILQ_INSERT_TAIL(&some_struct_g->list, l, linkage);
}
some_struct_g->a = 4;
some_struct_g->b = 5;
some_struct_g->n.c = 6;
some_struct_g->np = &some_struct_g->n;
some_struct_g->npp = &some_struct_g->np;
some_struct_g->nppp = &some_struct_g->npp;
some_struct_g->s = "kids";
for (z = 0; z < 10; ++z) {
papa.kids[z].a = 9 + z;
papa.kids[z].b = 9 + z;
papa.kids[z].n.c = 9 + z;
papa.kids[z].np = &papa.kids[z].n;
papa.kids[z].npp = &papa.kids[z].np;
papa.kids[z].nppp = &papa.kids[z].npp;
papa.kids[z].s = "papa";
}
]]--
local function attach_cb(bt)
-- Note: this is a global (Lua) variable.
-- cached_ref_var will be nil if this fails. variable_cb will
-- attempt to use this, resulting in an exception if nil.
cached_ref_var = pmodule.bt_query("some_struct_g", "crash", "crash.c")
local n = cached_ref_var.list.stqh_first
local v = pmodule.variable_from_bt(n)
while v:value() ~= 0 do
local val = pmodule.variable_from_bt(n.v)
print(string.format("[%x] %s: %d",
val:address(), val:name(), val:value()))
n = n.linkage.stqe_next
v = pmodule.variable_from_bt(n)
end
end
local function variable_cb(var, raw)
-- Variables synthesized this way will be finalized/freed. They cannot
-- be added to the snapshot. All synthesized variables must go through
-- the synthesis API.
-- This is enforced through the API.
local n = pmodule.variable_from_bt(raw.len):value()
local addr = pmodule.variable_from_bt(raw.kids):value()
-- We'll use size to iterate over the array.
local size = pmodule.sizeof(pmodule.deref(cached_ref_var))
print(string.format("addr: %x, len: %d, size: %d", addr, n, size))
for i = 0, n - 1 do
local na = addr + i * size
print(string.format("--- iter %d (%x)---", i, na))
elem = pmodule.deref(cached_ref_var, addr + i * size)
print(string.format("a: %d",
pmodule.variable_from_bt(elem.a):value()))
print(string.format("b: %f",
pmodule.variable_from_bt(elem.b):value()))
-- Note: when accessing member fields, pointers are
-- automatically dereferenced until we reach the underlying
-- struct.
print(string.format("nppp.c: %d",
pmodule.variable_from_bt(elem.nppp.c):value()))
end
end
local function pm_load()
local m = pmodule.match()
m:add_object("crash")
m:add_file("crash.c")
m:add_variable_base_type("array_struct", pmodule.match_type.substr)
m:add_variable_ptrace_type(pmodule.variable_type.tuple)
m:add_variable_name("papa", pmodule.match_type.substr)
pmodule.register(pmodule.event.variable, variable_cb, m)
pmodule.register(pmodule.event.attach, attach_cb)
end
pmodule.define{
id = "array_iter",
load = pm_load
}
There are a few CLI options to keep in mind when using pmodules:
- sandbox whitelist: Allow certain Lua features to be used by Lua modules (disabled by default for better default security).
- module-load/modules-path: Load modules from a specific file or directory.
- module: Pass key-value arguments to modules (modules are identified by their id name).
For more details, consult ptrace -h output.
That concludes this tutorial. We've reviewed how to register your pmodule with ptrace, how to register callbacks for events that interest you, how to iterate over various snapshot objects, and some of what you can do with individual objects. Of course, there's more one can do in a pmodule -- for a full reference, consult the pmodule API documentation.