# AI imports & sync

Source: https://docs.qiick.io/externa/ai-imports-and-sync

Beyond chat, the assistant can import collection rows from attachments or remote JSON, export data, extract PDF text, schedule sync sources, and roll back the last AI mutation turn. Async work is tracked as import jobs; an external webhook can enqueue the same pipeline with HMAC + `AI_WEBHOOK_TOKEN`.

> **Related**
> Chat UI & attachments: [AI chat](https://docs.qiick.io/externa/ai-chat). Tool registration gates: [AI assistant model](https://docs.qiick.io/externa/ai-assistant). Env keys: [AI configuration](https://docs.qiick.io/externa/ai-configuration). Routes: [AI API & webhooks](https://docs.qiick.io/externa/ai-api).

## Tools

Registered on `App\Ai\Agents\AppAssistant` when the user holds the matching permission (then re-checked inside the tool).

| Tool class            | Gate                            | Role                                                                                           |
| --------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------- |
| `ImportCollectionCsv` | `can-create-collections`        | CSV/TXT/XLSX attachment → collection; max 500 rows sync; async if `async` or rows > 200        |
| `ImportRemoteJson`    | `can-create-collections`        | Remote JSON URL via `SafeRemoteUrlValidator`; limit default 200, max 500; same async threshold |
| `ExportCollection`    | `can-show-collections`          | Export up to 500 rows as `csv` or `json`                                                       |
| `ExtractPdfText`      | `can-create-collections`        | PDF attachment text, max 20 000 characters                                                     |
| `RollbackLastAiTurn`  | `can-delete-collections`        | Soft-delete creations from the last AI turn (`ai_prompt` → following `ai_mutation`)            |
| `GetImportJobStatus`  | `can-create-collections`        | Poll job status (mirror of HTTP status)                                                        |
| `ManageAiSyncSources` | create vs edit collection perms | List/create/update/enable/disable/delete sync sources                                          |

Paths: `app/Ai/Tools/{ToolName}.php`.

### `dry_run` and `upsert_key`

Both `ImportCollectionCsv` and `ImportRemoteJson` accept:

| Option       | Meaning                                         |
| ------------ | ----------------------------------------------- |
| `dry_run`    | Preview inferred schema / sample; **no writes** |
| `upsert_key` | Field name used to choose update vs create      |

Also accepted on the collection-import webhook and stored on `AiSyncSource` for scheduled runs. `ImportCollectionJob` carries `upsertKey` and `dryRun` in its constructor.

### CMS connector profiles

Optional `profile` on `ImportRemoteJson` (and queued jobs):

| Value       | Effect                                                                                  |
| ----------- | --------------------------------------------------------------------------------------- |
| `directus`  | Strip Directus system audit fields; promote first `translations[]` scalars onto the row |
| `wordpress` | Unwrap `*.rendered` envelopes before flatten                                            |

Migration guide: [CMS import connectors](https://docs.qiick.io/externa/cms-import).

## Import job status

| Item             | Detail                                                                            |
| ---------------- | --------------------------------------------------------------------------------- |
| Job              | `App\Jobs\ImportCollectionJob`                                                    |
| HTTP             | `GET /ai/import-jobs/{jobId}` → `ImportJobStatusController`                       |
| Name             | `ai.import-jobs.show`                                                             |
| Extra middleware | `permission:can-create-collections` (in addition to `can-use-ai` on the group)    |
| Status source    | `ImportCollectionJob::status($jobId)` — cache key `ai:import-job:{uuid}`, TTL 24h |

Status payload includes `job_id`, `status` (`queued` \| `running` \| `done` \| `failed`), `processed`, `total`, `message`, and `result` (user id stripped for the response).

## Webhook

| Item         | Detail                                                      |
| ------------ | ----------------------------------------------------------- |
| Route        | `POST /ai/webhooks/collection-import`                       |
| Controller   | `App\Http\Controllers\Ai\CollectionImportWebhookController` |
| Name         | `ai.webhooks.collection-import`                             |
| Middleware   | **Only** `throttle:30,1` (no session auth)                  |
| Token config | `config('ai.webhook_token')` ← **`AI_WEBHOOK_TOKEN`**       |

Auth (from **1.2.2**, [#214](https://github.com/qiick-io/externa-core/pull/214); fail-closed every env in **1.3.0**, [#185](https://github.com/qiick-io/externa-core/issues/185) / [#207](https://github.com/qiick-io/externa-core/issues/207)): signature verified **before** validation. Header **`X-AI-Webhook-Signature`** = `sha256=` + HMAC-SHA256(**raw body only**, `AI_WEBHOOK_TOKEN`, plus grace previous token when set). Empty configured token → **401** in **every** environment (no local unsigned bypass); mismatch → **401**. Bearer alone rejected. Rotation: [AI API — Rotate](https://docs.qiick.io/externa/ai-api#rotate-the-webhook-token).

Body (`CollectionImportWebhookRequest`):

| Field           | Rules        |
| --------------- | ------------ |
| `collection_id` | Required     |
| `records`       | Array, 1–500 |
| `upsert_key`    | Optional     |

Uses `ImportsCollectionRecords`; webhook path does not write AI mutation logs (`logAiMutation` no-op).

## Scheduled sync (`AiSyncSource`)

| Item     | Detail                                                                                                                                  |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Model    | `App\Models\AiSyncSource`                                                                                                               |
| Fillable | `user_id`, `collection_id`, `url`, `upsert_key`, `interval_minutes`, `auth_bearer` (encrypted), `last_run_at`, `last_status`, `enabled` |
| Command  | `ai:run-sync-sources` → `App\Console\Commands\RunAiSyncSourcesCommand`                                                                  |
| Schedule | `Schedule::command('ai:run-sync-sources')->everyMinute()` in `routes/console.php`                                                       |

Eligible sources enqueue `ImportCollectionJob` with `sourceType: 'remote_json'` and `syncSourceId`.

## Remote URL safety

| Item      | Detail                                                                                           |
| --------- | ------------------------------------------------------------------------------------------------ |
| Validator | `App\Ai\Support\SafeRemoteUrlValidator`                                                          |
| Used by   | `ImportRemoteJson`, `ManageAiSyncSources`                                                        |
| Rules     | http/https only; no URL credentials; block localhost / metadata / private IPs; DNS resolve check |
| Allowlist | `config('ai.remote_import_hosts')` ← **`AI_REMOTE_IMPORT_HOSTS`** (comma-separated)              |

If the allowlist is non-empty, the host must be listed. If empty, any non-blocked host is allowed.

## Related env vars

| Env                      | Config key               |
| ------------------------ | ------------------------ |
| `AI_REMOTE_IMPORT_HOSTS` | `ai.remote_import_hosts` |
| `AI_WEBHOOK_TOKEN`       | `ai.webhook_token`       |
| `AI_DAILY_PROMPT_LIMIT`  | `ai.daily_prompt_limit`  |

(Plus provider keys documented under [AI configuration](https://docs.qiick.io/externa/ai-configuration).)

## Source map

| Concern           | Location                                                                                                                                  |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Tools             | `app/Ai/Tools/ImportCollectionCsv.php`, `ImportRemoteJson.php`, `ExportCollection.php`, `ExtractPdfText.php`, `RollbackLastAiTurn.php`, … |
| Job               | `app/Jobs/ImportCollectionJob.php`                                                                                                        |
| Webhook           | `app/Http/Controllers/Ai/CollectionImportWebhookController.php`                                                                           |
| Job status        | `app/Http/Controllers/Ai/ImportJobStatusController.php`                                                                                   |
| Sync command      | `app/Console/Commands/RunAiSyncSourcesCommand.php`                                                                                        |
| Validator         | `app/Ai/Support/SafeRemoteUrlValidator.php`                                                                                               |
| Routes / schedule | `routes/ai.php`, `routes/console.php`                                                                                                     |