A Pin Tool for tracing instructions specifically for tenet.
A special thank you to hasherezade's tiny_tracer of which this tool is generously built on.
# Remove any previous trace or metadata files before starting a new trace session
Remove-Item -Path trace.db, trace.db-shm, trace.db-wal, trace.meta.txt -ErrorAction SilentlyContinue; `
# Run Intel Pin with the Tenet Tracer tool.
# -o: Output prefix/directory for the trace
# -i: Specify rebase address for each image (case sensitive!)
# -w: Only log activity for these modules (must match on case)
# --: Arguments after this go to the target program
& "C:\pin\pin.exe" -t "C:\pin\source\tools\tenet_tracer\x64\Release\tenet_tracer.dll" `
-o F:\output\trace `
-i "notepad.exe:0x140000000" ` # Specify rebase address for notepad.exe
-i "ntdll.dll:0x180000000" ` # These are case sensitive!
-w "notepad.exe" ` # Only log notepad.exe activity
-w "ntdll.dll" `
-w "CoreMessagingXP.dll" ` # note the capitalization (it must match!)
-- C:\Windows\System32\notepad.exe- Input:
reg=0xval,reg=0xval,rip=0xpc,mr=0xaddr:hexbytes,mw=0xaddr:hexbytes - Stored:
timestamp(int), tid(int), pc(int), registers(json), mem_reads(json), mem_writes(json) - Output: Same as input (via
scripts/dump_trace_db.py)
rsp=0xdaf4efefd8,rip=0x7ff9339627a0,mr=0x7ff93481aa18:a0279633f97f0000,mw=0xdaf4efefd8:1b5c7b34f97f0000
rsp=0xdaf4efefa0,rip=0x7ff9339627a4
rip=0x7ff9339627a9,mw=0xdaf4efefc0:20867c34f97f0000
rdx=0x7ff9339b1730,rip=0x7ff9339627b0
rip=0x7ff9339627b5,mw=0xdaf4efefc8:0000000000000000
r8=0xdaf4efefc0,rip=0x7ff9339627ba
rsp=0xdaf4efefa0,rip=0x7ff9339627c1,mr=0xdaf4efef98:c1279633f97f0000
rip=0x7ff9339627c6
rcx=0x0,rip=0x7ff9339627c8
rip=0x7ff9339627ca
rcx=0x1,rip=0x7ff9339627cd
rax=0x1,rip=0x7ff9339627cf
rsp=0xdaf4efefd8,rip=0x7ff9339627d3
rsp=0xdaf4efe648,rip=0x7ff9339627a0,mr=0x7ff93481aa18:a0279633f97f0000,mw=0xdaf4efe648:1b5c7b34f97f0000Structured Schema:
-
Trace data is stored in dedicated, queryable columns:
timestamp(INTEGER): Unix epoch secondstid(INTEGER): Thread IDpc(INTEGER): Program counterregisters(TEXT): JSON-encoded register deltas, e.g.{"rax":"0x1234"}mem_reads(TEXT): JSON array of memory reads, e.g.[{"addr":"0xaddr","data":"hexbytes"}]mem_writes(TEXT): JSON array of memory writes
-
Table layout:
<prefix>_trace: Main trace for all threads<prefix>_trace_<tid>: Per-thread tables for parallel/filtered analysis
-
Indexing:
idx_<prefix>_trace_tid_tson(tid, timestamp)for fast time/range queries
Optimized Insertion:
-
Uses prepared statements (
sqlite3_prepare_v2,sqlite3_bind_*) for all inserts:- Prevents SQL injection
- Avoids quote/escape handling
- Boosts performance through statement reuse
-
Batch commits: Entries are accumulated (default batch size: 100), then written in a single transaction:
- Wraps inserts in
BEGIN ... COMMIT - Massively reduces I/O overhead
- Wraps inserts in
-
WAL mode is enabled for safe concurrent read access and improved durability:
PRAGMA journal_mode=WALPRAGMA synchronous=NORMAL
CREATE TABLE trace_trace (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp INTEGER, -- Unix epoch seconds
tid INTEGER, -- Thread ID
pc INTEGER, -- Program counter
registers TEXT, -- JSON: register deltas (i.e. {"rax":"0x1234"})
mem_reads TEXT, -- JSON: memory reads
-- (i.e. [{"addr":"0x...", "data":"..."}])
mem_writes TEXT -- JSON: memory writes
-- (i.e. [{"addr":"0x...", "data":"..."}])
);
CREATE INDEX idx_trace_trace_tid_ts ON trace_trace(tid, timestamp);python3 dump_trace_db.py trace.db --statspython3 dump_trace_db.py trace.db --listpython3 dump_trace_db.py trace.db --output trace.log
python3 dump_trace_db.py trace.db --tid 1234 --output thread_1234.logsqlite3 trace.db "SELECT COUNT(*) FROM trace_trace;"sqlite3 trace.db "SELECT tid, COUNT(*) FROM trace_trace GROUP BY tid;"sqlite3 trace.db "SELECT printf('0x%X', pc), COUNT(*) as cnt FROM trace_trace GROUP BY pc ORDER BY cnt DESC LIMIT 10;"sqlite3 trace.db "SELECT MIN(timestamp), MAX(timestamp), MAX(timestamp)-MIN(timestamp) as duration FROM trace_trace;"sqlite3 trace.db "SELECT * FROM trace_trace WHERE mem_writes != '[]' LIMIT 10;"sqlite3 trace.db "SELECT * FROM trace_trace WHERE pc >= 0x400000 AND pc < 0x500000;"sqlite3 -csv -header trace.db "SELECT timestamp, printf('0x%X',pc) as pc, registers FROM trace_trace" > trace.csvFind RAX changes
sqlite3 trace.db "SELECT * FROM trace_trace WHERE json_extract(registers, '$.rax') IS NOT NULL LIMIT 10;"Extract register value
sqlite3 trace.db "SELECT json_extract(registers, '$.rax') FROM trace_trace WHERE json_extract(registers, '$.rax') IS NOT NULL LIMIT 5;"- Use per-thread tables (trace_trace_
<tid>) for better parallelism - Query with tid in WHERE clause to use index
- Batch size 100-500 optimal for most cases
- Use LIMIT when exploring large traces
- Sample with "WHERE id % N = 0" for overview
# ifdef USE_SQLITE3
sqlite3* db = nullptr;
sqlite3_open("trace.db", &db);
auto logger = tenet_tracer::LoggerBuilder()
.addHandler(std::make_unique<tenet_tracer::SqliteLogHandler>(db, "trace", 100))
.build();
logger->setThreadId(tid);
logger->log("rax=0x1234,rip=0x400000");
logger->close();
sqlite3_close(db);
# endif-
Trace entries not appearing?
- Batches may not be flushed yet. Call
logger->close()or ensure program shutdown triggers a flush.
- Batches may not be flushed yet. Call
-
"Database locked" errors?
- WAL mode is enabled by default; if you encounter lock errors, double-check your SQLite configuration.
-
Slow queries?
- Use the
tid, timestampindex. Add further indexes based on query needs.
- Use the
-
Want even faster write performance?
- Increase batch size (e.g. 500–1000). For maximum speed at the risk of corruption, try
PRAGMA synchronous=OFF.
- Increase batch size (e.g. 500–1000). For maximum speed at the risk of corruption, try
-
Format compatibility concerns?
- Use
dump_trace_db.pyto convert SQLite data back to the original text format. Note: PC field (rip/eip) may differ by architecture, but both refer to the program counter.
- Use
-
"Database locked" errors?
- WAL mode is enabled by default; double-check for multi-process access or database handles left open.
-
Empty query results?
- Entries may still be in the queue. Call
logger->close()or let the program fully exit to flush all data.
- Entries may still be in the queue. Call
-
Slow queries?
- Use
LIMIT, filter bytid, and make sure indexes exist for your queries.
- Use
-
Wrong output format?
- Use
dump_trace_db.pyto convert SQLite data back to the original format.
- Use
-
High memory usage?
- Lower the
batch_sizeargument in the handler constructor to reduce queue growth.
- Lower the
This project will not work if you use Windows* (MSVC). This is CRITICALLY IMPORTANT!
To compile the prepared project you need to use Visual Studio 2022.
It was tested with Intel Pin 3.31's Windows* (LLVM clang-cl).
Why?
Because Intel Pin Tools for LLVM's clang-cl support all of C++11 and some extras support some of C++14. This allows us to write Pin code in MODERN C++.
Now, just clone this repo into \source\tools that is inside your Pin root directory. Open the project in Visual Studio and build.
- In order for Pin to work correctly, Kernel Debugging must be DISABLED.
- In
install32_64you can find a utility that checks if Kernel Debugger is disabled (kdb_check.exe, source), and it is used by the Tenet Tracer's.batscripts. This utilty sometimes gets flagged as a malware by Windows Defender (it is a known false positive). If you encounter this issue, you may need to exclude the installation directory from Windows Defender scans.
- Normalize registers and memory to separate tables
- Store memory data in binary (not hex)
- Compress JSON columns for storage savings
- Periodically
VACUUMto reclaim space - Support live analysis via a read-only DB while tracing is active
- Streaming analytics (aggregations) during trace capture