Skip to content
Shape

Generate Drafts from Source

shp ast source parses source files with Tree-sitter and prints a Shape draft of their structure. The draft is candidate context for review, not a reviewed claim: every generated function keeps effects unknown, and the claims a team maintains belong in authored .shape files.

Source parsing supports TypeScript (.ts, .mts, .cts), TSX (.tsx), JavaScript and JSX (.js, .jsx, .mjs, .cjs, all parsed with the JavaScript grammar), Rust (.rs), Go (.go), Python (.py), and Swift (.swift). --language overrides the extension; the CLI Reference lists its values and the other flags. Release archives bundle the parsers beside shp, so generation downloads nothing. The bundled native parser runs on Linux x64 and arm64 with glibc, macOS arm64, and Windows x64. Windows archives include a native binding that loads inside the compiled executable without a separate Node.js installation. On musl Linux, generation fails with unsupported_tree_sitter_platform.

What a draft contains

Section titled “What a draft contains”

The semantic draft maps code structure onto Shape declarations:

  • Components. A type becomes a component when it owns functions (methods, impl blocks, or Go receiver methods) or has a field that looks like state: repo, client, store, db, database, queue, sender, receiver, connection, or pool. A file’s free functions go into a <File>Module component.
  • Functions. Each function becomes an fn under its owner, with a source reference and effects unknown. A reference names the declaration’s symbol, such as src/history/log.rs#RevisionLog.append_revision, or only the file when no stable symbol exists, so moving lines does not change the draft.
  • Resources. A type that is not a component becomes a resource only when its name ends in Event, Record, Message, Config, State, Entity, Snapshot, Request, Response, Table, Queue, Topic, or Payload. Any other such type is left out.
  • Calls. A call through self., this., or a Go receiver to a field whose declared type is another generated component becomes a relation of kind calls. Calls that cannot be resolved that way are left out.
  • Implementations. Each component gets an implementation <Name>Impl listing its source file. It has no on_change clause, so it never governs paths for coverage.
  • AST anchors. Each generated type, resource, and function gets an AST anchor: a resource with the GeneratedAstAnchor trait whose storage ast.anchor("path#symbol") names one piece of syntax. A relation of kind generated_from connects the declaration (for a function, its component) to the AST anchor.
  • Marker traits. Each draft declares two empty marker traits: GeneratedCandidate, on every generated component and resource, and GeneratedAstAnchor, on every AST anchor. The checker gives neither any meaning.
  • Fingerprints. An AST anchor carries a fingerprint, a hash of its syntax. The provider ast.semantic_subtree_v1 hashes a canonical form of the anchored node’s subtree: node kinds, field names, child order, attributes, labels, and token text such as identifiers, literals, operators, modifiers, and keywords. It excludes file paths, spans, node IDs, comments, and whitespace, so reformatting, comment edits, and line movement leave it unchanged. A function anchor covers the whole function, body included. A type anchor keeps only the signatures of the functions nested in it, so editing a method body changes the method’s fingerprint but not the type’s. The generated_from relation expects the AST anchor’s current fingerprint. An expects ... fingerprint line in a relation, or a pin line in a candidate effect, is a pin: shp check compares it with the AST anchor’s current fingerprint.
  • Candidate effects. An effect candidate records a guess, not a claim. It is emitted when a function’s name or body contains a keyword and the body mentions a generated resource. The first matching group wins: append, add, insert, create, push, write, save, or persist give Append; then update, set, mutate, or replace give Update; then delete, remove, purge, clear, or drop give Delete; then read, get, load, fetch, find, list, or query give Read. It has confidence low and pins the function’s anchor fingerprint. shp check validates a candidate’s fields, names, and pin, reporting stale candidate effect pin when the pin differs from the anchor, but never treats it as a declared effect.

A generated draft

Section titled “A generated draft”

This Rust file is the input:

pub struct RevisionRecord {
    pub id: String,
}


pub struct RevisionRepo;


impl RevisionRepo {
    pub fn insert(&self, _revision: RevisionRecord) {}
}


pub struct RevisionLog {
    repo: RevisionRepo,
}


impl RevisionLog {
    pub fn append_revision(&self, revision: RevisionRecord) {
        self.repo.insert(revision);
    }
}

shp ast source src/history/log.rs prints the draft to stdout under the default module name generated.ast; --module sets another. The excerpt below keeps the declarations for RevisionLog.append_revision. It omits the RevisionRepo declarations and the type anchors for RevisionRecord and RevisionLog, which follow the same pattern.

module generated.ast


trait GeneratedAstAnchor {
}


trait GeneratedCandidate {
}


resource RevisionLogAppendRevisionAstAnchor : GeneratedAstAnchor {
  storage ast.anchor("src/history/log.rs#RevisionLog.append_revision")
  fingerprint ast.semantic_subtree_v1("sha256:47c0534f294562bb62d23df8208cb048e3a31760723740372e25ada530f5f4fe")
}


resource RevisionRecord : GeneratedCandidate {
  storage rust.type("src/history/log.rs#RevisionRecord")
}


component RevisionLog : GeneratedCandidate {
  fn append_revision
    source rust("src/history/log.rs#RevisionLog.append_revision")
    effects unknown
}


relation RevisionLogAppendRevisionGeneratedFromRevisionLogAppendRevisionAstAnchor {
  kind generated_from
  connects RevisionLog -> RevisionLogAppendRevisionAstAnchor
  roles { RevisionLog as generated, RevisionLogAppendRevisionAstAnchor as syntax }
  expects RevisionLogAppendRevisionAstAnchor fingerprint ast.semantic_subtree_v1("sha256:47c0534f294562bb62d23df8208cb048e3a31760723740372e25ada530f5f4fe")
  summary "fn RevisionLog.append_revision generated from rust function_item at src/history/log.rs#RevisionLog.append_revision."
}


relation RevisionLogCallsRevisionRepo {
  kind calls
  connects RevisionLog -> RevisionRepo
  roles { RevisionLog as origin, RevisionRepo as destination }
  summary "RevisionLog.append_revision calls RevisionRepo; generated from src/history/log.rs#RevisionLog.append_revision."
}


effect candidate AppendRevisionAppendRevisionRecordCandidateEffect {
  fn RevisionLog.append_revision
  effect Append<RevisionRecord>
  source rust("src/history/log.rs#RevisionLog.append_revision")
  confidence low
  pin RevisionLogAppendRevisionAstAnchor fingerprint ast.semantic_subtree_v1("sha256:47c0534f294562bb62d23df8208cb048e3a31760723740372e25ada530f5f4fe")
}


implementation RevisionLogImpl {
  paths {
    "src/history/log.rs"
  }
  conforms_to RevisionLog
}

RevisionLog is a component because it has a method and a repo field. RevisionRecord has no functions and a data-like name, so it becomes a resource. The calls relation comes from self.repo.insert(revision), and the candidate effect from the word insert next to a mention of RevisionRecord.

Commit generated context

Section titled “Commit generated context”

--out-dir writes one generated file per source file, plus a manifest:

$ shp ast source --out-dir shape/generated/ast src/history/log.rs
Wrote 1 generated AST Shape file(s) to shape/generated/ast.

The draft lands in shape/generated/ast/src/history/log.shape under the module shape.generated.ast.src.history.log, with the same anchors and fingerprints as the stdout draft. shape/generated/ast/manifest.json records which files the generator owns:

{
  "version": 1,
  "generatedAt": "deterministic",
  "entries": [
    {
      "module": "shape.generated.ast.src.history.log",
      "path": "shape/generated/ast/src/history/log.shape",
      "sources": [
        "src/history/log.rs"
      ]
    }
  ]
}

These rules shape the output tree:

  • Each source path is made relative to the workspace root (the git top-level, otherwise the working directory) before the module name, output path, and source references are derived. An absolute path, or a run from a nested directory, produces the same module names and source references. A source outside the workspace is rejected.
  • The module name is the module base (default shape.generated.ast; --module may name a child of it) followed by the source path without its extension, one segment per directory, with characters other than letters, digits, and _ replaced by _. When two sources map to the same module name, the later source, in path order, gets a deterministic _g suffix of eight hex digits. Two sources that map to the same output file, such as foo.ts and foo.tsx, are rejected before anything is written.
  • Regeneration rewrites every file the manifest owns and removes owned files that are no longer produced. Other files in the tree, such as authored .shape files, are left alone. Hand edits to generated files do not survive regeneration.

Default discovery loads the generated files with the rest of the Shape model. Strict shp check accepts their effects unknown through a narrow exemption: the unknown-effects diagnostic is suppressed only for a function whose module is shape.generated.ast or starts with shape.generated.ast., and whose file lies under shape/generated/ast/ relative to the repository root (by default, the working directory). Both conditions are required, and the manifest plays no part. The same draft saved anywhere else fails with error: unknown effects, as does effects unknown in any authored module; see Effect Model. The language server does not apply this exemption, so an editor running shp lsp shows error: unknown effects in generated files that shp check accepts. Generated functions also never count as a Shape update for coverage; see Keep the Model Current.

In CI, run the same command with --check and the same source list. It regenerates in memory, compares the result with the files the manifest owns, writes nothing, and exits 1 when any owned file is stale, missing, or no longer produced:

$ shp ast source --out-dir shape/generated/ast --check src/history/log.rs
error: generated AST Shape files are stale


  shape/generated/ast/src/history/log.shape

When everything matches, it prints Generated AST Shape files are up to date in shape/generated/ast. and exits 0.

Promote and pin reviewed claims

Section titled “Promote and pin reviewed claims”

A reviewed claim goes in an authored module. The overlay below imports the generated module, declares the reviewed effect, and pins the syntax it reviewed:

module history


import shape.generated.ast.src.history.log


resource RevisionRecord : AppendOnly


component RevisionLog {
  owns RevisionRecord
  grants Append<RevisionRecord>
  fn append_revision
    source rust("src/history/log.rs#RevisionLog.append_revision")
    effects complete {
      Append<RevisionRecord>
        evidence rust("src/history/log.rs#RevisionLog.append_revision")
    }
}


relation RevisionLogAppendRevisionReviewedFromAst {
  kind generated_from
  connects RevisionLog -> shape.generated.ast.src.history.log::RevisionLogAppendRevisionAstAnchor
  roles { RevisionLog as reviewed, shape.generated.ast.src.history.log::RevisionLogAppendRevisionAstAnchor as syntax }
  expects shape.generated.ast.src.history.log::RevisionLogAppendRevisionAstAnchor fingerprint ast.semantic_subtree_v1("sha256:47c0534f294562bb62d23df8208cb048e3a31760723740372e25ada530f5f4fe")
  summary "Reviewed RevisionLog.append_revision effects are backed by the generated anchor for src/history/log.rs#RevisionLog.append_revision."
}

RevisionLog and RevisionRecord here are the authored declarations in module history; a local declaration wins over an imported one of the same name. The generated anchor is named with its module qualifier. With the generated file in place, shp check passes.

The pin must live in the authored file. Regeneration rewrites every generated file, including the generated generated_from relation’s expects, so a generated pin always matches and cannot detect change. The authored expects stays fixed while the anchor it names is regenerated. When the body of append_revision changes and the context is regenerated, shp check fails with exit 1:

error: stale fingerprint expectation


relation RevisionLogAppendRevisionReviewedFromAst expects RevisionLogAppendRevisionAstAnchor fingerprint ast.semantic_subtree_v1.
expected: sha256:47c0534f294562bb62d23df8208cb048e3a31760723740372e25ada530f5f4fe
actual: sha256:178f10c360afabebe17e02cd913fe98750d3fb81829647aea70550ec4212b6b8


caused by:
  - shape/history.shape: relation RevisionLogAppendRevisionReviewedFromAst expects RevisionLogAppendRevisionAstAnchor fingerprint ast.semantic_subtree_v1
  - shape/generated/ast/src/history/log.shape: resource RevisionLogAppendRevisionAstAnchor
  - shape/generated/ast/src/history/log.shape: resource RevisionLogAppendRevisionAstAnchor fingerprint ast.semantic_subtree_v1

When the function is renamed or removed, regeneration drops its anchor, and the authored relation names an endpoint that no longer exists:

error: unknown relation_endpoint


relation_endpoint RevisionLogAppendRevisionAstAnchor is referenced but not declared.


caused by:
  - shape/history.shape: relation RevisionLogAppendRevisionReviewedFromAst

Either failure means the reviewed syntax changed: review the function again, update the claim, and replace the pinned fingerprint with the actual: value from the diagnostic, or the endpoint with the new AST anchor. A change to comments or whitespace alone keeps the pin valid.

Missing token evidence

Section titled “Missing token evidence”

A fingerprint needs token text in the anchored subtree. When a declaration has none, generation keeps the draft and exits 0, but prints a warning on stderr. The anchor is emitted without a fingerprint, its generated_from relation has no expects, and any candidate effect that would pin it is skipped, so no claim can be pinned to that anchor. This mostly happens with shp ast json input that omits text. For example, JSON input whose RevisionRecord struct node has no text produces:

warning missing_fingerprint_tokens: src/history/log.rs:src_history_log_rs_log_struct_item_1bc227f3fc917ba9: cannot compute ast.semantic_subtree_v1 for RevisionRecord; skipping fingerprint because AST JSON/source node lacks token text or semantic child tokens

Adapters and the raw layer

Section titled “Adapters and the raw layer”

shp ast json builds the same draft from syntax another tool has already parsed. It reads exactly one JSON file; Shape never exports AST JSON. The input has this contract:

  • The top level has files (required) and an optional language.
  • Each file has path, root (the ID of its root node), and nodes, a flat list. It may set its own language. The language comes from the file, then the top level, then the path’s extension; with none of these, generation fails.
  • Each node has id and kind. It may have text, attributes, children, named (default true), span, and textHash. children lists child IDs, as strings or as { "id": ..., "field": ... } objects.
  • attributes is a flat map of strings, numbers, booleans, and null. Nested structure belongs in child nodes; a nested attribute is rejected.
  • Declarations are recognised by Tree-sitter kind names. Types are struct_item, enum_item, class_declaration, class_definition, interface_declaration, type_declaration, and type_spec; implementation blocks are impl_item and implementation_item; functions are function_item, function_declaration, function_definition, method_definition, and method_declaration. Nodes of other kinds produce no declarations; they still count toward fingerprints and appear in the raw layer.
  • A declaration’s name comes from attributes.name, then from a name field child or an identifier child, then from the node’s text.
  • text also feeds the heuristics: a type becomes a component through a state-like field only when its text shows that field.
  • The input is rejected for duplicate node IDs, a missing root or child, a node with two parents, a cycle, or a node unreachable from its root.
{
  "language": "rust",
  "files": [
    {
      "path": "src/history/log.rs",
      "root": "root",
      "nodes": [
        { "id": "root", "kind": "source_file", "children": ["log"] },
        {
          "id": "log",
          "kind": "struct_item",
          "attributes": { "name": "RevisionLog" },
          "text": "struct RevisionLog { repo: RevisionRepo }"
        }
      ]
    }
  ]
}

shp ast json --module generated.history ast.json turns this into a component with its anchor. This excerpt omits the generated_from relation and the implementation:

module generated.history


trait GeneratedAstAnchor {
}


trait GeneratedCandidate {
}


resource RevisionLogAstAnchor : GeneratedAstAnchor {
  storage ast.anchor("src/history/log.rs#RevisionLog")
  fingerprint ast.semantic_subtree_v1("sha256:93620cec0ac0a519f118c514ffd6eec44c4a9c898c781a8b7ceabb8a7bded130")
}


component RevisionLog : GeneratedCandidate {
}

The raw layer records every syntax node, which helps when debugging an adapter or preserving exact parser provenance. --include-ast-layer adds it to the stdout draft; --raw-out PATH writes it to a separate file under the module <module>.raw while stdout keeps the semantic draft. Each file and node becomes a generated resource with storage ast.file(...) or storage ast.node(...), and each parent-child edge becomes a relation of kind ast_child. The 19-line Rust file above yields 100 raw resources. The two options exclude each other, and neither combines with --out-dir.

Swift notes

Section titled “Swift notes”

Swift drafts come from syntax alone; no Swift compiler, Xcode, or SwiftPM configuration is loaded.

  • Classes, structs, enums, actors, protocols, and extensions supply types. Functions, protocol requirements, initializers, deinitializers, subscripts, and computed properties supply functions, including a SwiftUI view’s computed body even when the view has no other methods. Stored properties are part of the type’s syntax. Local functions stay inside their enclosing function’s syntax, and types declared inside functions, with anything nested in them, produce nothing.
  • A Swift type becomes a component unless it is a single declaration with no functions, no state-like field, and a data-like name (the resource suffixes above), in which case it becomes a resource.
  • Within a file, extensions are grouped with their type and keep separate anchors. Nested types use qualified names such as Outer.Inner.run().
  • References include parameter labels and types, generic parameters, and return and effect syntax, such as RevisionLog.load(id:Int) -> Revision ? or RevisionLog.save(_:Revision) async throws. A constrained extension keeps its where clause, and a protocol requirement ends in [requirement]. These are review conventions, not compiler symbol IDs. Ambiguous duplicate signatures produce a warning and file-only references.
  • Every Swift function keeps effects unknown, and Swift drafts contain no inferred calls or candidate effects. Macros are not expanded, #if branches are not chosen, protocol and generic dispatch are not resolved, actor isolation is not proven, and types are not merged across files.
  • Fingerprints use parser tokens, so literal URLs, raw and multiline strings, interpolation, and Unicode are part of the evidence, while comments and formatting outside literals are not. Type anchors exclude method and computed-property bodies; function anchors include them.

An authored Swift claim uses the same reference syntax:

module swift.history


resource Revision : AppendOnly


component RevisionLog {
  owns Revision
  grants Append<Revision>
  fn save
    source swift("Sources/RevisionLog.swift#RevisionLog.save(_:Revision) async throws")
    effects complete {
      Append<Revision>
        evidence swift("Sources/RevisionLog.swift#RevisionLog.save(_:Revision) async throws")
    }
}