Read handlers

Building a connector, step 7 (HTTP track, 3 of 5). Previous: Authentication and configuration.

Read handlers are what Sync calls: list every record of an entity, read one record, and list the relationships that become Grants. Each handler is a module whose default export takes its input first and a context object, ctx, last. It returns plain values and throws on failure.

How a handler looks

import { isNotFound } from "@owlie/connectors/sdk/v2";
import type { Handler, UserRecord } from "@connector/types";
import { resolveUserId } from "@connector/lib/lookups.ts";

const fetchOne: Handler<"user.fetchOne"> = async (identifiers, ctx) => {
  try {
    const id = await resolveUserId(ctx, identifiers);
    const response = await ctx.http.get<UserRecord>(`/users/${encodeURIComponent(id)}`, ctx.options);
    return {
      record: response.data,
      metadata: { lifecycleState: response.data.status === "deactivated" ? "DISABLED" : "ACTIVE" },
    };
  } catch (error) {
    if (isNotFound(error)) return { record: null };
    throw error;
  }
};

export default fetchOne;

isNotFound comes from @owlie/connectors/sdk/v2, the only package a connector may import besides its own files. Those it imports as @connector/types and @connector/lib/<file>.ts, which read the same from every folder; relative paths such as ../../lib/lookups.ts still work. identifiers holds whichever of the entity's identifiers the caller has, so each one is optional in the generated type: resolve them in one helper in lib/ (here resolveUserId, which uses the id when present and otherwise looks the user up by email, failing with NOT_FOUND when nobody matches). Handler<"user.fetchOne"> types the input and the return value from your manifest; a key your manifest does not route to does not exist in the type. The Connector Builder's editor type-checks as you type. Owlie's build does not: a type error does not stop a test or a publish, but a syntax error or an import that does not resolve does.

The context

Member	Use
ctx.http	The HTTP client: get, post, put, patch, delete. Authentication and the base URL are already applied, and a non-2xx response throws a classified error.
ctx.paginate	Reads one page of a list call (below).
ctx.options	The caller's cancellation signal and timeout. Spread it into every ctx.http call.
ctx.config	The install's configuration, read-only: ctx.config.config holds the shared settings, ctx.config.auth the chosen option's key and config.
ctx.validate	Checks a record against its entity schema.
ctx.errors	Errors to raise yourself (Provider errors and quirks).
ctx.log	A structured logger (debug, info, warn, error).

Every member is described in Handler API.

Always pass ctx.options: ctx.http.get(path, ctx.options), or { ...ctx.options, query: { ... } } when you add options of your own. Without it, a cancelled sync keeps waiting on your request.

Read one record

<entity>.fetchOne receives the entity's identifiers and returns { record } with the system's raw record, or { record: null } when it does not exist. Provisioning calls it to read back what it changed. For the account entity, add metadata.lifecycleState (ACTIVE, DISABLED or DELETED) mapped from the system's status. Never include Grants in fetchOne: they come from Sync.

List records

<entity>.fetchAll returns one page: the records, the cursor for the next page and whether there are more. ctx.paginate does this for most list calls in one line:

const fetchAll: Handler<"user.fetchAll"> = (options, ctx) => ctx.paginate<UserRecord>({ path: "/users", options });

It uses the manifest's pagination_type unless you pass style. The styles:

Style	The API pages with	Default parameters
cursor	A cursor in the body	cursor and limit; next cursor read from next_cursor, nextCursor or next
next-token	A token in the body or a header	pageToken and pageSize; next token from nextPageToken, next_page_token or next_token
offset-limit	An offset and a page size	offset and limit; total from total
page-number	A page number	page and per_page, starting at page 1
link-header	A Link: <...>; rel="next" header	per_page

Override what differs: params (parameter names), items (the body key holding the records; by default the body itself when it is an array, else the first array under items, data, results, records or value), next (where the next token is), total, firstPage, pageSize ({ default, max }), query and headers. A next value that is a full URL is followed as a URL.

ctx.paginate stops with an error rather than guessing, because a sync that silently ends early looks like every remaining record was deleted:

• a response with no item array, or items that are not objects, is an error, never an empty page;
• a page or position the walk saw among its last eight is an error that names it;
• for offset-limit and page-number without a total, only an empty page ends the walk; declare fullPages: true if the API always fills a page, so a short page ends it;
• for offset-limit and page-number with a total, the walk counts the records actually read, and an empty page before the total is an error;
• a next URL on a different origin is refused.

APIs that page in a way these styles cannot express (a cursor in a POST body, a GraphQL pageInfo, a nested next link) are handled in your own code with ctx.http: return { records, nextCursor, hasMore } yourself.

Return one page per call. Owlie calls fetchAll again with the cursor you returned, and in Owlie's cloud each handler run has its own budget of CPU time and outbound requests (Handler API), so a handler that walks every page in one run fails on a large system.

Check records against the schema with ctx.validate.resourceSchema("user", record, { mode: "warn" }). Use warn for data the system returns: a response that drifts from your schema is logged, not failed.

Changes since the last sync

If the entity declares sync_support.supports_incremental: true, write <entity>.fetchAllSince as well, and return a marker on every page of both fetchAll and fetchAllSince. The marker is yours: a timestamp, a sequence number, or an object. It means "everything up to what I have returned so far". Owlie saves the latest one and passes it back as options.marker on the next run, and fetchAllSince must return what changed after it, with no gaps.

Incremental runs return only records that changed, so relationships that arrive embedded on those records are seen only when the system updates the record's own change marker. A relationship removed from an unchanged record disappears at the next full run.

List relationships

Each Grant kind is read in one of two ways, set by its fetch_mode:

• per-entity: Sync calls <entity>.fetchRelationships on the Grant kind's discovered_via entity, once per record of that entity, with that record's external id.
• embedded: the relationships come back with the entity's own records, in the relationships array of the page fetchAll returns. Always include the array, empty when the page's records carry none: provisioning reads Grants back from these pages and fails on a page that has records but no relationships array.

Either way, each relationship is a record like this:

{
  "edgeKey": "team_member",
  "sourceEntityKey": "user",
  "sourceExternalId": "u_1042",
  "targetEntityKey": "team",
  "targetExternalId": "t_design",
  "attrs": { "role": "admin" }
}

edgeKey is the Grant kind's key. The external ids are each entity's first identifier, the same value its own records carry. Qualifier values ride in attrs under each qualifier's name (or the path its source names). A fetchRelationships handler for Acme Projects' teams:

import type { Handler } from "../../types.gen.ts";

const fetchRelationships: Handler<"team.fetchRelationships"> = async (teamId, options, ctx) => {
  const page = await ctx.paginate<{ user_id: string; role: string }>({
    path: `/teams/${encodeURIComponent(teamId)}/members`,
    options,
  });
  return {
    ...page,
    records: page.records.map((member) => ({
      edgeKey: "team_member",
      sourceEntityKey: "user",
      sourceExternalId: member.user_id,
      targetEntityKey: "team",
      targetExternalId: teamId,
      attrs: { role: member.role },
    })),
  };
};

export default fetchRelationships;

Per-entity relationships are always read in full; there is no marker on fetchRelationships. A relationship Sync no longer sees at the end of a complete run is removed. If a record refers to its target by something that can change (a path or a name), resolve it to the target's id before returning the relationship, and fail the page when a current reference does not resolve.

Next: Write handlers