Flat-file definition

Every field of a flat-file connector's definition.json, the declaration Owlie derives from it, how uploaded files are routed and read, and the rules Owlie checks. A flat-file definition lists feeds: which uploaded file holds which entity or relationship, how it is parsed, and which column is which. The machine-readable form is flat-file.schema.json.

Editable in the tables says whether a value can be install-editable: written as { "$oxl": "config.<name>" }, naming a declared config field whose default is the seed every install copies and may then edit. Only two values can be: a feed's fileMatch and an entity feed's columns.

A complete definition is in Example: CSV feed.

Top level

Field	Type	Required	What it does
engine	integer	yes	Always 1, the version of the definition format.
description	string	no	What the catalog shows. Non-empty.
icon	string	no	The catalog icon: @ plus a domain, or an image URL. Non-empty.
config	object	no	The install's configuration fields. Its one key is properties.
config.properties	object of JSON Schemas	no	One entry per config field, keyed by name. In practice, the seeds of install-editable values.
accountEntityKey	string	no	The entity whose records are accounts: the entityKey of one of the entity feeds. Without it, no entity is the account.
feeds	object	yes	The feeds, keyed by feed name. A feed name cannot contain /.

A config field is required in the install form unless its default is "" or null. A field that an install-editable value reads must have the matching type: string for fileMatch, object for columns.

Feed fields

Every feed has these fields:

Field	Type	Required	Editable	What it does
kind	entity or relationship	yes	no	What the feed's rows are: records of one entity, or links between two entities' records.
fileMatch	string	yes	yes	The file name pattern that routes an uploaded file to the feed.
parse	object	yes	no	How the feed's files are read.
parse.delimiter	",", ";" or "\t"	yes	no	The cell delimiter: comma, semicolon or tab.
parse.header	boolean	yes	no	Whether the first row names the columns.
parse.encoding	"utf-8"	yes	no	The file encoding. UTF-8 is the only one.

Entity feeds

A feed with "kind": "entity" also has:

Field	Type	Required	Editable	What it does
entityKey	string	yes	no	The entity the rows are records of. One feed per entity.
columns	object	yes	yes	The columns, keyed by column name. At least one column.
columns.<name>.position	integer, 0 or more	yes	with columns	The column's position, 0 for the first. Unique within the feed.
columns.<name>.type	string, number or boolean	yes	with columns	How the column's cells are converted.
columns.<name>.identifier	boolean	no	seeded identifiers stay fixed	Whether the column identifies a record.
isActiveExpr	string	no	no	On the account entity's feed only: an OXL expression over the record, true when the account is active.
isDeletedExpr	string	no	no	On the account entity's feed only: an OXL expression over the record, true when the account is deleted.

An entity feed needs at least one identifier column and may have several. The identifier with the lowest position is the record's external id. Owlie takes it from the first identifier column, in position order, that has a value in the row, so a row whose first identifier is blank is keyed by the next one; a row with every identifier blank is a failed record. When two rows share an external id, the later row wins. The expressions read the record's columns by name, such as employment_status == 'active'.

Relationship feeds

A feed with "kind": "relationship" also has:

Field	Type	Required	What it does
edge	object	yes	The relationship the rows describe.
edge.edgeKey	string	yes	The relationship's name. Unique among the relationship feeds.
edge.sourceEntityKey	string	yes	The entity each row links from: an entity one of the entity feeds provides.
edge.targetEntityKey	string	yes	The entity each row links to: an entity one of the entity feeds provides.
edge.sourceColumn	string	yes	The header name of the column that holds the source record's external id.
edge.targetColumn	string	yes	The header name of the column that holds the target record's external id.

A relationship feed requires parse.header: true. It has no entityKey, no install-editable columns, and no isActiveExpr or isDeletedExpr. Its edge.sourceColumn must be a column of the source entity's feed, and its edge.targetColumn a column of the target entity's feed, in each install's columns.

How files are routed and read

Step	Rule
Routing	An upload that names a feed goes to it. Otherwise the file goes to the first feed, in definition order, whose fileMatch matches the file name, using each install's own patterns. In a pattern, * matches any run of characters and ? one character; case is ignored. No match refuses the upload.
Size	A file larger than 8 MiB fails ingestion.
Parsing	Cells split on parse.delimiter. A quoted cell may hold the delimiter, doubled quotes and line breaks. Blank lines are skipped.
Columns, with a header	Each cell goes to the column whose name is its header text exactly. A header column with no declared column is kept on the record as text.
Columns, without a header	Cells go to the declared columns in position order: the first cell to the lowest position, and so on.
string	The cell's text as it is.
number	The number the trimmed cell holds, or the text when it holds none. A blank cell is null.
boolean	true, 1, false or 0, trimmed, in any case. A blank cell is null. Any other value makes the row unreadable.
Relationship rows	Each row links the record whose external id is under edge.sourceColumn to the record whose external id is under edge.targetColumn. An id no record has yet gets a placeholder record, which the entity file fills in when it brings that record. A blank end, or a header without either column, fails the run.
Lifecycle	For each account record, isDeletedExpr giving true makes it deleted; otherwise isActiveExpr giving true makes it active and false disabled. Any other result sets no state.

Each file is its feed's complete snapshot. A successful run removes the records, or relationships, that earlier files contained and this file does not. A run that finds no records after a previous successful run found some fails and removes nothing. A run with unreadable rows, or rows with no identifier value, ingests the rest and skips every removal; a file where unreadable rows pile up fails the run.

Rules Owlie checks

Owlie refuses a definition, or an install's configuration, that breaks any of these:

• engine is 1 and feeds is an object.
• description and icon, when present, are non-empty literal strings.
• A feed name has no /, kind is entity or relationship, and fileMatch is a non-empty string. No two feeds have the same fileMatch, compared exactly (routing ignores case, so patterns that differ only in case are accepted and the first one takes the files).
• An entity feed has an entityKey no other feed has, and no edge. Its columns are an object of at least one column, each with a position that is a non-negative integer no other column has, and a type of string, number or boolean. At least one column is an identifier.
• isActiveExpr and isDeletedExpr appear only on the feed whose entityKey is accountEntityKey.
• A relationship feed has parse.header: true, no entityKey, no install-editable columns, and an edge with an edgeKey no other relationship feed has. edge.sourceEntityKey and edge.targetEntityKey are entities that entity feeds provide.
• accountEntityKey, when present, is an entity that an entity feed provides.
• An install-editable value is exactly { "$oxl": "config.<name>" }, naming a declared config field of the matching type.
• An install's own columns keep every seeded identifier column with the same name, type and position, still marked identifier, and mark no other column as an identifier.

The derived declaration

Part	Derived value
provides	["sync"]: no provisioning.
protocols	["s3"]: files arrive in an Owlie-hosted drop.
auth	One option, { "key": "none" }: installs enter no credentials.
config	config.properties, each required unless its default is "" or null.
Entities	One per entity feed, keyed by its entityKey.
Grant kinds	None. Relationship feeds are not part of the declaration; their rows are read when their files are ingested.

Each entity has:

• name: its entityKey, and the description Flat-file feed "<feed name>";
• is_account: true for the entity named by accountEntityKey;
• sync_support: never incremental;
• a record schema with one property per column, of the column's type, identifier columns marked x-identifier, and other properties allowed;
• identifier_paths: the identifier columns in position order.

The account entity's feed passes isActiveExpr and isDeletedExpr to its sync job as the account's lifecycle expressions. Every feed's sync job has no schedule: it runs when a file arrives, or by hand.

Operations

Handler	What it does
connection.validateConfig	Derives the feeds with the install's values, and reports any rule they break.
connection.testConnection	Always succeeds: the drop is Owlie's.

There are no other handlers. Owlie supplies both, so a flat-file connector has nothing to override.