> For the complete documentation index, see [llms.txt](https://docs.jobfront.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.jobfront.io/jobs-data-file-exports/data-exports.md).

# Data Exports

This document describes the JobFront collections export system — how job data is packaged and delivered to customers, what fields are included in each format, how often exports refresh, and how to configure custom field mappings.

***

### Overview

The JobFront collections export system continuously publishes job data from your configured collections to your chosen delivery destination. Each collection maps to a curated set of employer sources; the export system packages all active jobs from those sources into structured files and pushes them to you on a scheduled cadence.

Three output formats are supported:

| Format      | Best for                                                             |
| ----------- | -------------------------------------------------------------------- |
| **JSON**    | REST API consumers, data pipelines, modern integrations              |
| **XML**     | ATS integrations, legacy job board ingestion, feed-based systems     |
| **Parquet** | Data warehouses, analytics platforms, bulk ML/data science workloads |

***

### Export Cadence & Frequency

| Setting                     | Value         |
| --------------------------- | ------------- |
| Default refresh interval    | Every 6 hours |
| Configurable per collection | Yes           |

Collections are evaluated in a continuous scheduling loop. Each collection has its own independent timer. When a collection's configured interval has elapsed since its last successful export, a new export run begins immediately.

***

### Delivery Methods

#### S3 (Jobfront-hosted)

Files are written to a designated S3 bucket and folder path. You access them via:

* **Pre-signed URLs** — time-limited download links (default 1-hour TTL) provided by the Jobfront API. No AWS credentials required on your end.
* **IAM credentials** — a read-only AWS IAM user scoped to your prefix. Use standard AWS CLI or SDK tooling to sync files directly.

```bash
# AWS CLI example — download all files for your organization
aws s3 sync s3://<bucket>/<your-prefix>/ ./local-data/ \
  --exclude "*" --include "*.json" --include "*.xml" --include "*.parquet"
```

```python
# Python (boto3 + pyarrow) — read a Parquet file directly without downloading
import pyarrow.parquet as pq
import s3fs

fs = s3fs.S3FileSystem(key='YOUR_KEY', secret='YOUR_SECRET')
df = pq.read_table('s3://<bucket>/<your-prefix>/jobs.parquet', filesystem=fs).to_pandas()
```

#### SFTP (push to your server)

Files are pushed directly to your SFTP server after each successful export. Configuration requires:

* Hostname
* Username / authentication credentials
* Base directory path
* Optional connection timeout (default: 30 seconds)

***

### File Organization

Each collection can be configured in one of two file organization modes:

| Mode                    | Description                                                                  | Typical use case                                         |
| ----------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------- |
| **`aggregate_sources`** | All employers in the collection are merged into a single file per export run | Feed to a single job board or ATS                        |
| **`single_source`**     | One separate file is produced per employer/source                            | Per-employer intake pipelines, employer-specific routing |

In `single_source` mode the filename includes the source identifier so each employer's file can be processed independently.

***

### Export Formats

#### JSON & XML Exports

Four named format variants are available. All variants expose the same fields — the choice controls the output structure and element naming conventions:

| Format identifier | Type | Description                   |
| ----------------- | ---- | ----------------------------- |
| `json_jobfront`   | JSON | Standard Jobfront JSON schema |
| `xml_jobfront`    | XML  | Standard Jobfront XML schema  |

The fields included in each export are controlled by the **data tier** setting (`min`, `default`, or `max`) described in the next section.

#### Parquet Exports

Parquet files use [Apache Parquet](https://parquet.apache.org/) format with **Snappy compression** and a row group size of 50,000 rows. The schema is fixed (see Parquet Schema below). Parquet exports always include the equivalent of the `default` tier fields plus additional timestamp and lifecycle metadata.

Each export cycle produces:

```
jobs-active-{N}.parquet                       — job data (N = 0 through 39)
metrics/jobs-active-latest-metrics-{N}.json   — row counts, source counts, timing
metrics/jobs-active-latest-job_ids-{N}.json   — list of job IDs in this file
metrics/jobs-active-latest-source_ids-{N}.json — list of source IDs in this file
```

***

### Data Tiers (min / default / max)

JSON and XML exports support three data tiers. Higher tiers are supersets — each tier includes all fields from the tiers below it.

#### min tier

The core required fields present in every export.

| Field                 | Type          | Description                                                                                                                   |
| --------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `id`                  | string        | Unique Jobfront job identifier                                                                                                |
| `title`               | string        | Job title                                                                                                                     |
| `description`         | string        | AI-generated summary of the job post                                                                                          |
| `post`                | string (HTML) | Full job post content                                                                                                         |
| `url_job`             | string        | Direct application / posting URL                                                                                              |
| `job_status`          | string        | `active` or `inactive`                                                                                                        |
| `created`             | string        | Date first seen, formatted `YYYY-MM-DD`                                                                                       |
| `created_at`          | integer       | Unix timestamp when job was first seen                                                                                        |
| `salary.min`          | float \| null | Minimum salary. Null if not available. If only one bound is listed in the source, both min and max are set to the same value. |
| `salary.max`          | float \| null | Maximum salary. Null if not available.                                                                                        |
| `salary.currency`     | string        | ISO currency code, e.g. `USD`                                                                                                 |
| `salary.period`       | string        | `annual`, `hourly`, `monthly`, or `weekly`                                                                                    |
| `onet_code`           | string        | O\*NET 6-digit occupation code, e.g. `15-1252.00`                                                                             |
| `remote`              | boolean       | Whether the position is fully remote                                                                                          |
| `locations[].text`    | string        | Full location text string                                                                                                     |
| `locations[].street`  | string        | Street address                                                                                                                |
| `locations[].city`    | string        | City / locality                                                                                                               |
| `locations[].state`   | string        | State / region                                                                                                                |
| `locations[].country` | string        | Country                                                                                                                       |
| `locations[].zip`     | string        | ZIP / postal code                                                                                                             |
| `source.id`           | string        | Internal employer/source identifier                                                                                           |
| `source.name`         | string        | Employer / company name                                                                                                       |

#### default tier

Includes all `min` fields, plus:

| Field                     | Type            | Description                                                               |
| ------------------------- | --------------- | ------------------------------------------------------------------------- |
| `work_model`              | string          | `on_site`, `remote`, or `hybrid`                                          |
| `commitment`              | string          | Employment type: `full_time`, `part_time`, `contract`, `internship`, etc. |
| `level`                   | string          | Seniority level: `junior`, `mid`, `senior`, `executive`, etc.             |
| `responsibilities`        | array of string | Responsibilities extracted from the job post                              |
| `post_language`           | string          | BCP-47 language code of the job post, e.g. `en`                           |
| `jobboard_format`         | string          | Source job board type/format                                              |
| `offers_visa_sponsorship` | boolean         | Whether the post mentions visa sponsorship                                |
| `source.domain`           | string          | Employer website domain                                                   |
| `source.url_logo`         | string          | URL to the employer's logo image                                          |

#### max tier

Includes all `default` fields, plus:

| Field                  | Type            | Description                                                               |
| ---------------------- | --------------- | ------------------------------------------------------------------------- |
| `benefits`             | array of string | Benefits mentioned in the post (health, PTO, 401k, etc.)                  |
| `requirements`         | array of string | Requirements and qualifications listed                                    |
| `problems`             | array of string | Problem statements extracted from the post                                |
| `categories`           | array of string | Job category classifications                                              |
| `technologies`         | array of string | Technologies and skills inferred from the post                            |
| `locations[].lat`      | float           | Latitude (geo-coordinate)                                                 |
| `locations[].lon`      | float           | Longitude (geo-coordinate)                                                |
| `verified_inactive_at` | integer \| null | Unix timestamp when the job was confirmed inactive. Null if still active. |
| `scraped_extracted.*`  | object          | Raw fields extracted directly by the scraper                              |
| `brands`               | array of string | Brand names extracted from the post                                       |
| `addresses`            | array of string | Full address strings extracted from the post                              |
| `source.description`   | string          | Employer description                                                      |
| `source.industries`    | array of string | Industry tags for the employer                                            |
| `source.tags`          | array of string | Freeform tags for the employer                                            |

***

### Data Quality Filters

Every job included in an export — regardless of format — must pass export-time quality checks.

**Deduplication** is applied within each export run:

* By `job_id` — a job cannot appear twice in the same file
* By compound key `(job_board_source_id, source_id)` — prevents the same logical posting from appearing under multiple job board IDs

***

### Custom Field Mapping System

By default, exports use the JobFront field names and structure documented above. The optional custom mapping system lets you define a JSON spec that transforms every job record into a completely different shape before it is written to the output file — renaming fields, restructuring nested objects, applying value maps, computing derived strings, and more.

A custom mapping is stored per organization. The active mapping is the most recently published spec for your `organization_id`.

#### How Mappings Work

When a mapping spec is configured, each job record passes through the mapping engine after enrichment but before serialization. The engine walks the `fields` object in your spec and builds a new output record from scratch. Fields not referenced in your spec are omitted from the output.

The mapping engine operates on the **`default`-tier** job record (the full enriched job object), so all fields documented in the `default` and `max` tiers are available as source paths.

#### Field Spec Reference

Each key in `fields` is the output field name; the value is a spec object describing how to produce it.

***

**`from` — map a source field directly**

```json
{ "title": { "from": "title" } }
{ "apply_url": { "from": "url_job" } }
{ "city": { "from": "locations[0].city" } }
```

Supports dot-path notation and array indexing, e.g. `locations[0].city`.

***

**`fromAny` — first non-empty wins**

```json
{
  "compensation_min": {
    "fromAny": ["salary.min", "scraped_extracted.compensation_min"],
    "default": null
  }
}
```

Tries each path in order and uses the first one that is present and non-empty.

***

**`const` — fixed value**

```json
{ "country": { "const": "USA" } }
{ "feed_version": { "const": "2" } }
```

***

**`default` — fallback value**

Used with `from` or `fromAny` when the source field is missing or empty.

```json
{ "level": { "from": "level", "default": "mid" } }
```

***

**`template` — compose a string from multiple fields**

```json
{ "location_text": { "template": "{city}, {state}, {country}" } }
{ "salary_range": { "template": "{salary_min}–{salary_max} {salary_currency}" } }
```

Uses `{field_name}` placeholders resolved from the source record. If all referenced fields are empty the output is an empty string.

Use `[[ ... ]]` for conditional segments — the segment is omitted entirely if all its fields are empty:

```json
{ "display_title": { "template": "{title}[[ — {level}]]" } }
```

***

**`map` — output a nested object**

```json
{
  "location": {
    "map": {
      "city":    { "from": "locations[0].city" },
      "state":   { "from": "locations[0].state", "transform": "state_abbr" },
      "country": { "from": "locations[0].country", "transform": "country_code" }
    }
  }
}
```

***

**`listMap` — transform a list of objects**

```json
{
  "skills": {
    "from": "technologies",
    "listMap": {
      "skill_name": { "from": "name", "transform": "lower" }
    }
  }
}
```

Each item in the source list is run through the `listMap` sub-spec.

***

**`listMapPrimitive` — transform a list of primitives to a list of objects**

```json
{
  "skill_objects": {
    "from": "inferred_skills",
    "listMapPrimitive": {
      "name": { "fromSelf": true, "transform": "lower" }
    }
  }
}
```

Use `fromSelf: true` to reference the primitive item itself.

***

**`buildList` — conditionally assemble a list**

```json
{
  "tags": {
    "buildList": [
      { "value": "remote",   "when": { "equals": ["remote", true] } },
      { "from":  "level",    "when": { "exists": "level" } },
      { "from":  "commitment" }
    ]
  }
}
```

Each entry is evaluated in order; entries whose `when` condition fails are skipped.

***

**`wrapAsList` and `splitBy` — scalar to list**

```json
{ "category_list": { "from": "categories[0]", "wrapAsList": true } }
{ "tag_list": { "from": "source.tags", "wrapAsList": true, "splitBy": "," } }
```

`splitBy` splits a delimited string into an array before wrapping.

***

**`joinWith` — list to string**

```json
{ "skills_csv": { "from": "technologies", "joinWith": ", " } }
```

***

**`wrapWith` — add prefix / suffix**

```json
{ "job_ref": { "from": "id", "wrapWith": { "prefix": "JOB-", "suffix": "" } } }
```

***

**`number` and `boolean` — type coercion**

```json
{ "salary_min_num": { "from": "salary.min", "number": true } }
{ "is_remote_bool": { "from": "remote", "boolean": true } }
```

***

**`nullIfEmpty` and `nullIfZero` — output null instead of blank / zero**

```json
{ "salary_min": { "from": "salary.min", "nullIfZero": true } }
{ "department":  { "from": "department",  "nullIfEmpty": true } }
```

***

**`regexReplace` — pattern-based string rewriting**

A list of rules applied in order. Each rule specifies `pattern`, `replacement`, and optional `ignoreCase`.

```json
{
  "clean_title": {
    "from": "title",
    "regexReplace": [
      { "pattern": "\\(remote\\)",     "replacement": "", "ignoreCase": true },
      { "pattern": "\\s{2,}",          "replacement": " " }
    ]
  }
}
```

***

**`setUrlParams` — inject or replace URL query parameters**

```json
{
  "tracking_url": {
    "from": "url_job",
    "setUrlParams": { "utm_source": "jobfront", "utm_medium": "feed" }
  }
}
```

***

**`valueMap` — remap specific values**

```json
{
  "employment_type": {
    "from": "commitment",
    "valueMap": {
      "full_time":  "FT",
      "part_time":  "PT",
      "contract":   "CT",
      "internship": "IN"
    }
  }
}
```

Any value not present in the map is passed through unchanged.

***

#### Transform Reference

The `transform` key applies a single named transformation to the value after it is resolved.

| Value               | Effect                                                                             |
| ------------------- | ---------------------------------------------------------------------------------- |
| `lower`             | Converts string to lowercase                                                       |
| `upper`             | Converts string to uppercase                                                       |
| `title`             | Title-cases the string                                                             |
| `trim`              | Strips leading and trailing whitespace                                             |
| `state_abbr`        | Normalizes US state name to 2-letter abbreviation (e.g. `California` → `CA`)       |
| `country_code`      | Normalizes common US country variants to `USA`                                     |
| `numeric_rightmost` | Extracts the rightmost numeric component from a hyphen/underscore-delimited string |
| `numeric_leftmost`  | Extracts the leftmost numeric component from a hyphen/underscore-delimited string  |

***

#### Date Format Reference

The `dateFormat` key converts timestamp values. An optional `timezone` key (IANA tz name, e.g. `America/New_York`) applies timezone-aware conversion.

| Value        | Input                    | Output example                  |
| ------------ | ------------------------ | ------------------------------- |
| `unix->iso`  | Unix timestamp (integer) | `2024-06-15T09:30:00`           |
| `unix->date` | Unix timestamp (integer) | `2024-06-15`                    |
| `unix->rfc`  | Unix timestamp (integer) | `Sat, 15 Jun 2024 09:30:00 EDT` |
| `iso->date`  | ISO datetime string      | `2024-06-15`                    |

```json
{ "posted_date": { "from": "created_at", "dateFormat": "unix->date" } }
{
  "posted_datetime_et": {
    "from": "created_at",
    "dateFormat": "unix->iso",
    "timezone": "America/New_York"
  }
}
```

***

#### Conditional Logic

The `cases` key evaluates a list of conditions and uses the first match.

```json
{
  "seniority": {
    "cases": [
      { "when": { "equals":   ["level", "senior"] }, "const": "Senior" },
      { "when": { "equals":   ["level", "junior"] }, "const": "Junior" },
      { "when": { "exists":   "level"              }, "from":  "level"  }
    ],
    "default": "Mid"
  }
}
```

**Supported `when` operators:**

| Operator      | Syntax                                           | Description                                                        |
| ------------- | ------------------------------------------------ | ------------------------------------------------------------------ |
| `exists`      | `{ "exists": "field.path" }`                     | True if the field is present and non-null                          |
| `equals`      | `{ "equals": ["field.path", value] }`            | True if the field equals the given value                           |
| `contains`    | `{ "contains": ["field.path", "substring"] }`    | True if the string field contains the substring (case-insensitive) |
| `notContains` | `{ "notContains": ["field.path", "substring"] }` | True if the string field does not contain the substring            |
| `equalsField` | `{ "equalsField": ["field.a", "field.b"] }`      | True if two fields have equal values                               |

***

#### Example Mapping Spec

```json
{
  "fields": {
    "JobId":          { "from": "id" },
    "JobTitle":       { "from": "title" },
    "JobDescription": { "from": "description" },
    "FullPost":       { "from": "post" },
    "ApplyUrl":       { "from": "url_job" },
    "PostedDate":     { "from": "created_at", "dateFormat": "unix->date" },
    "CompanyName":    { "from": "source.name" },
    "CompanyDomain":  { "from": "source.domain" },
    "IsRemote":       { "from": "remote", "boolean": true },
    "WorkModel":      { "from": "work_model" },
    "EmploymentType": {
      "from": "commitment",
      "valueMap": {
        "full_time":  "Full-Time",
        "part_time":  "Part-Time",
        "contract":   "Contract",
        "internship": "Internship"
      }
    },
    "SeniorityLevel": {
      "from": "level",
      "transform": "title",
      "default": "Not specified"
    },
    "SalaryMin":      { "from": "salary.min",      "number": true, "nullIfZero": true },
    "SalaryMax":      { "from": "salary.max",      "number": true, "nullIfZero": true },
    "SalaryCurrency": { "from": "salary.currency", "transform": "upper" },
    "SalaryPeriod":   { "from": "salary.period" },
    "OnetCode":       { "from": "onet_code" },
    "City":           { "from": "locations[0].city" },
    "State":          { "from": "locations[0].state", "transform": "state_abbr" },
    "Country":        { "from": "locations[0].country", "transform": "country_code" },
    "LocationText":   { "template": "{city}, {state}" },
    "Skills":         {
      "from": "technologies",
      "listMap": { "name": { "from": "name" } }
    },
    "TrackingUrl": {
      "from": "url_job",
      "setUrlParams": { "utm_source": "jobfront", "utm_medium": "feed" }
    }
  }
}
```

***

### Export Run Metadata

Each export run is recorded with the following fields, accessible via the Jobfront API or metrics endpoints:

| Field                          | Description                                                                           |
| ------------------------------ | ------------------------------------------------------------------------------------- |
| `run_id`                       | Unique identifier for this export run                                                 |
| `organization_id`              | Your organization identifier                                                          |
| `collection_id`                | Identifier of the collection exported                                                 |
| `export_type`                  | Delivery method: `s3_jobfront` or `sftp_customer`                                     |
| `export_data_format`           | Format: `json_jobfront`, `json_aspen`, `xml_jobfront`, `xml_aspen`                    |
| `export_data_type`             | Data tier: `min`, `default`, or `max`                                                 |
| `export_sources_file_fan_type` | File organization: `aggregate_sources` or `single_source`                             |
| `count_total_jobs`             | Total jobs included in this export                                                    |
| `count_total_sources`          | Total employer sources included in this export                                        |
| `export_time_start_at`         | Unix timestamp — when the export run began                                            |
| `export_time_end_at`           | Unix timestamp — when the export run completed                                        |
| `export_time_duration`         | Duration in seconds                                                                   |
| `export_url_file`              | URL to the exported file                                                              |
| `export_url_metrics`           | URL to the per-run metrics JSON                                                       |
| `track_metrics_dictionary`     | Summary metrics: total jobs/sources, most recent job added/updated/removed timestamps |
| `track_qa_dictionary`          | Field fill-rate statistics — what percentage of jobs have each field populated        |

***

### Location Filtering

Exports can be filtered by geography. Location filters are configured per collection and applied before export:

| Filter                          | Description                                                                                      |
| ------------------------------- | ------------------------------------------------------------------------------------------------ |
| `country`                       | Limit to jobs in a specific country (default: United States)                                     |
| `state`                         | Limit to jobs in a specific US state                                                             |
| `city`                          | Limit to jobs in a specific city                                                                 |
| `remote`                        | Include or exclude remote jobs                                                                   |
| `allow_remote_without_location` | When `true`, remote jobs with no location data are included regardless of other location filters |

Remote jobs (`is_remote = true`) pass through country/state/city filters by default — they are not excluded by geography unless `remote: false` is explicitly set.

***

### **Recency Filtering**

Exports can be filtered by job age. Recency filters are configured per collection and applied before export. All keys are optional — if no recency filter is set, jobs of any age are included.

| Filter            | Description                                                                                                                             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `after_ts`        | Include only jobs created at or after this Unix-second timestamp (fixed lower bound)                                                    |
| `before_ts`       | Include only jobs created at or before this Unix-second timestamp (fixed upper bound)                                                   |
| `newer_than_days` | Include only jobs created within the last N days (rolling lower bound, re-evaluated on every export run)                                |
| `older_than_days` | Include only jobs created more than N days ago (rolling upper bound, re-evaluated on every export run)                                  |
| `window`          | Preset shortcut for `newer_than_days`. Accepted values: `7d`, `30d`, `60d`, `90d`, `180d`, `365d`, `1y` (= 365 days), `2y` (= 730 days) |

Combine a "newer" bound (`after_ts`, `newer_than_days`, or `window`) with an "older" bound (`before_ts` or `older_than_days`) to export an age slice — for example, `newer_than_days: 365` with `older_than_days: 90` returns jobs between 90 and 365 days old.

If more than one key is provided for the same bound, explicit timestamps take precedence over relative values: `after_ts` wins over `newer_than_days` and `window`; `before_ts` wins over `older_than_days`.

Jobs with a missing or invalid `created_at` are excluded whenever any recency filter is active.

***

### **Job Title Filtering**

Exports can be filtered by job title text. Title filters are configured per collection and applied before export. All keys are optional — if no title filter is set, jobs with any title are included.

* Matching is case-insensitive substring. `include_contains: ["engineer"]` accepts both "Software Engineer" and "Engineering Manager".
* `include_contains` requires that the job title contain at least one of the listed terms. `exclude_contains` requires that the job title contain none of the listed terms.
* The two lists combine as a logical AND: with both set, the title must match at least one include term AND not match any exclude term — for example, `include_contains: ["engineer"]` with `exclude_contains: ["intern"]` includes "Software Engineer" and excludes "Engineering Intern".
* Each list is capped at 100 entries. Additional entries beyond the cap are silently dropped at parse time.
* Jobs with a missing or empty `job_title` are excluded whenever `include_contains` is set.

***

### **ONET Code Filtering**

Exports can be filtered by ONET occupation code. ONET filters are configured per collection and applied before export. All keys are optional — if no ONET filter is set, jobs with any ONET code are included.

* Codes that include a decimal (e.g., `15-1252.00`) match the job's exact ONET6 code. Codes without a decimal (e.g., `15-1252`) match every variant of that base code (`15-1252.00`, `15-1252.01`, `15-1252.07`, etc.) — useful for capturing every role under a single occupation group.
* `include_codes` requires that the job's ONET code match at least one entry. `exclude_codes` requires that the job's ONET code match none.
* The two lists combine as a logical AND: with both set, the job's ONET code must match an include AND not match an exclude — for example, `include_codes: ["15-1252"]` with `exclude_codes: ["15-1252.03"]` includes every Software Developers variant except the specific `15-1252.03` code.
* Exclude beats include: if a job's code matches both lists (e.g., a base include and an exact exclude target the same code), the job is dropped.
* A bare base code on the job side (e.g., `15-1252`) is treated as its canonical `.00` variant for exact-match comparisons, so `include_codes: ["15-1252.00"]` accepts both `15-1252.00` and a raw `15-1252`.
* Each list is capped at 100 entries. Additional entries beyond the cap are silently dropped at parse time.
* Jobs with a missing or empty ONET code are excluded whenever `include_codes` is set.