Upload files and run ingestion
Building a connector, step 7 (Flat-file track, 2 of 2). Previous: Define flat-file feeds.
Files reach a flat-file integration through its drop, a storage location Owlie hosts for that integration. A script or scheduled job uploads each export there with an upload token, and Owlie ingests every file as soon as it arrives. Each file is one feed's complete snapshot: what it contains is what Owlie keeps, and what it leaves out, Owlie removes.
This page describes an installed integration. You publish and install a flat-file connector like any other, as the next page shows.
Before the first upload
Each feed has its own sync job. Owlie creates the jobs from the feeds when the integration is installed with its sync jobs, and brings them up to date whenever the integration's configuration is saved. A file that arrives for a feed without a job fails, with a message asking you to save the integration's configuration; saving it creates the missing jobs, and then you upload the file again.
These jobs have no schedule. A feed's job runs when a file arrives for it, or when someone runs it by hand.
Create an upload token
An upload token lets a client request upload URLs for one integration, and nothing else. It does not
expire; revoke it when the client no longer needs it. Creating and revoking tokens needs the
integration.modify permission.
In Owlie
- Open Integrations, then the integration, then its Connection tab.
- In the Upload tokens card, enter a label that says which job uses the token, such as
payroll-export job, and select Create token. - Copy the token that appears. Owlie shows it only once and stores only a hash of it; the list then shows each token's label and last four characters.
Select Revoke beside a token to stop it working. The card also shows the integration's upload endpoint, which contains the integration's id.
Through the API
Three operations on POST /api/rpc manage tokens. Authenticate with an API key. The request body is
{ "procedure": "<name>", "input": { ... } }, and the response carries the procedure's output as
{ "result": { ... } }.
| Procedure | Input | Returns |
|---|---|---|
integrations.createUploadToken | { "integrationId": "...", "label": "payroll-export job" } (label optional) | { "ok": true, "token": "owlie_sk_oui...", "tokenId": "...", "tokenHint": "..." }. token is returned only here. |
integrations.listUploadTokens | { "integrationId": "..." } | tokens: each with id, label, tokenHint, createdAt and revokedAt |
integrations.revokeUploadToken | { "tokenId": "..." } | { "ok": true } |
A request looks like this, with an API key whose service account has the permission:
curl -X POST "https://acme.owlie.com/api/rpc" \
-H "Authorization: Bearer $OWLIE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "procedure": "integrations.createUploadToken", "input": { "integrationId": "<integration id>", "label": "payroll-export job" } }'
Build with the API and agents explains API keys and the endpoints.
Upload a file
An upload is two requests: ask Owlie for an upload URL with the token, then send the file to that URL.
1. Request an upload URL
curl -X POST "https://acme.owlie.com/api/integrations/$INTEGRATION_ID/upload-url" \
-H "Authorization: Bearer $OWLIE_UPLOAD_TOKEN" \
-H "Content-Type: application/json" \
-d '{ "filename": "employees_2026-09-27.csv" }'
Send it to your Owlie address, the one you sign in at (acme.owlie.com above), with the token as a
bearer token. The body names the file, and optionally the feed:
| Field | Required | What it does |
|---|---|---|
filename | yes | The file's name. Without feed, Owlie routes the file to the first feed, in definition order, whose file name pattern matches it, using the integration's own patterns. |
feed | no | The feed to route to, by feed name. It wins over the file name. |
filename and feed can also be sent as query parameters instead of a JSON body.
A successful request answers 200:
{
"url": "https://...",
"expiresAt": "2026-09-27T06:15:00.000Z",
"dropId": "4d8f0c2e-6a1b-4f5e-9c3d-2b7a8e1f0a64"
}
| Field | Meaning |
|---|---|
url | Where to upload the file. It works for 15 minutes and needs no other credential. |
expiresAt | When url stops working. |
dropId | The drop Owlie recorded for this upload, which you can find in the integration's drops. |
The request fails with:
| Status | error | Meaning |
|---|---|---|
401 | missing_upload_token | No Authorization: Bearer header. |
401 | invalid_upload_token | The token is wrong, revoked, or belongs to another integration. |
422 | filename_required | The request has no filename. |
404 | a message | The integration no longer exists. |
422 | no_matching_feed | No feed has that name, or no feed's pattern matches the file name. feeds lists the integration's feed names. |
429 | rate_limit_exceeded | More than 20 URL requests from your address in a minute. Wait and retry. |
2. Upload the file
curl -T employees_2026-09-27.csv "$UPLOAD_URL"
Send the file with an HTTP PUT to url, as curl -T does, with no other headers required. The file
must be UTF-8, in the feed's format, and at most 8 MiB: Owlie refuses to ingest a larger file.
Upload from the dashboard
To load a file by hand, use the File drops card on the integration's Connection tab. You need
the integration.modify permission. Choose Match by file name or a feed, then choose a .csv,
.tsv or .txt file of up to 8 MiB. Owlie ingests it as soon as the upload finishes, exactly like a
file uploaded with a token. If no feed matches, the card lists each feed with its pattern. If the
feed has no sync job yet, the upload is refused and asks you to save the integration's configuration
first.
What happens next
- The file arrives in the drop. Owlie moves it to the integration's archive.
- Owlie starts a full run of the feed's sync job, linked to that drop.
- The run reads the archived file with the feed's parse options and the integration's own columns, and brings the feed's records, or its relationships, up to date.
The File drops card lists the 50 newest drops with their File, Feed, Status, Size and Uploaded time, refreshing every few seconds. A drop's status:
| Status | Meaning |
|---|---|
UPLOADED | Owlie issued an upload URL for it. The file may not have arrived yet. |
ARCHIVED | The file arrived and is archived. Its run is starting or in progress, or has not started yet. |
INGESTED | The run finished and the file's contents are in Owlie. |
FAILED | The drop could not be ingested. The card shows the reason beside the status. |
Running the feed's sync job by hand retries the newest file for that feed that is archived but not yet ingested, or whose run failed or was canceled. It does not compare that file with newer ones: if a newer file has already been ingested, a run by hand still applies the older file, and removes what only the newer file added. Upload the current export again instead of retrying an old one. A run by hand with no such file fails and asks for an upload.
A drop that failed because its feed had no sync job never entered a run, so a run by hand does not retry it. Save the integration's configuration, which creates the missing jobs, then upload the file again.
Through the API, the integrations.listFileDrops procedure on POST /api/rpc returns the newest
drops in result.drops. Its input is { "integrationId": "...", "limit": 50 }, with limit from 1 to 200 (default 50),
and each drop has id, feed, originalFilename, status, sizeBytes, error, syncJobRunId and
createdAt. The run itself is on the Sync Runs page like any other.
Each file is a full snapshot
A successful run makes the integration's data for that feed match the file. A record, or a
relationship, that an earlier file contained and the new file does not is removed from the
integration, exactly as when Sync no longer finds a record in a connected system. There is no partial
or incremental upload: to remove someone, upload a file without them, or mark them with the account
feed's isDeletedExpr. Always upload the whole export, never only what changed.
Safeguards
Because a bad file could otherwise remove everything, Owlie holds back removals in these cases:
- An empty file. A file with no records, after the previous successful run found some, fails the run and removes nothing. The first file a feed receives may be empty.
- Unreadable rows. When any row of an entity feed's file cannot be read, such as a
booleancell that is nottrue,false,1or0, Owlie ingests the other rows but skips every removal for that run, because a row it could not read would otherwise look absent. The run is marked Removals skipped. The next clean file removes what is missing. - Rows without an identifier. A row whose identifier cells are all blank has no external id. It counts as a failed record, so it also skips removals for that run.
- Mostly unreadable files. A file where unreadable rows pile up (more than 100 of them among the rows Owlie reads at a time) fails the run instead, as the wrong file or the wrong format.
A relationship file with a blank end in any row, or without the sourceColumn or targetColumn
header, fails the run. A file larger than 8 MiB fails it too.
Test a flat-file connector
A flat-file connector has no operations that reach a system, so the Test panel has only the two connection checks: validating a configuration re-derives the feeds with the values you give it, and the connection test always succeeds. Test the definition and a real upload instead:
- Check the derivation. The Derived manifest tab shows each entity, its identifiers and the install fields. Nothing in the Problems tab should remain.
- Check the columns against a real export. After From a sample file or Infer columns from a
sample, each feed's Columns section previews the sample's first rows read with the feed's
columns, as ingestion will read them. It marks each cell that will not fit: a number column cell
that is not a number (kept as text), a boolean cell that is not
true,false,1or0(the row is rejected), a blank identifier, and a header column that no column names (kept as text). The preview stays until you reload the page or select Dismiss. - Upload a sample on a test integration. Publish, install the connector as a separate test
integration, save its configuration, create an upload token and upload a small file per feed.
Check that each drop reaches
INGESTEDand that the records and relationships appear on the integration as you expect. Then upload a second file with one record left out, and check that the record is removed.