How it’s made

Where the data comes from, every rule applied to it, what it can’t tell you, and how the numbers are checked.

The pipeline

flowchart LR
  A["Adzuna API<br/>16 searches, every morning"] --> B[("Raw ads<br/>JSON as received")]
  G["Internet Vacancy Index<br/>official, monthly"] --> H[("Raw official series")]
  B --> C["Clean ads<br/>one row per ad"]
  C --> D["Summaries<br/>one per question"]
  H --> D
  D --> E["This website"]
  D --> F["CSV export +<br/>independent check in Python"]

Every morning at 06:00 (Brisbane) a script runs 16 searches on the Adzuna API, each one matching words in the job title, and asks for the ads posted since the last successful run: normally the last two days, and up to 30 if earlier runs failed. Each ad is stored in PostgreSQL with the full record Adzuna sends; when the same ad comes back on a later day, its newest version replaces the old one. SQL then builds a clean table (one row per ad) and one summary per question on this site. The whole rebuild runs as a single transaction with 13 checks. If one of the 10 hard checks fails, nothing changes and yesterday’s numbers stay up; the other 3 only raise a warning. Once a week, the official Internet Vacancy Index is checked for a new month.

The rules

Rule
Role family From the job title, first match wins: noise (data centre jobs, lab and nursing roles) → Data Scientist → Data Engineer → BI → Business Analyst → Data Analyst → Data Leadership → Other. The family is the area; seniority is a separate field
Level Also from the title: Junior (graduate, intern, entry…) → Director → Manager → Lead → Senior → Not stated. Nothing is guessed
Duplicates Same title, company and place, posted at most 7 days apart: counted once
City From Adzuna’s location (“Melbourne Region” → Melbourne). Ads that only say a state or “Australia” stay as No city
Salary Annual, daily and hourly figures are kept apart and never converted into each other

A finding about salaries

Adzuna doesn’t say whether a salary is per year, day or hour, so the unit comes from the size of the number. A first rule read “180 to 240” as dollars per hour and made data engineers look like they earned AUD 210 an hour. Those figures were really thousands, but they weren’t salaries either: most of them start at exactly AUD 180,000, across more than a hundred employers, and include data centre technicians. They are a salary band set by the job board, not what the employer offers. They are kept, counted below, and left out of every salary figure.

Data quality, today

What this data can’t tell you

  • Which tools employers ask for (SQL, Python, Power BI…). Adzuna shares only the first ~500 characters of each ad, and the requirements usually come later.
  • Remote work, visas or security clearance: the share that mentions them early is shown as a floor, never as a measure.
  • The whole market. Adzuna aggregates many job boards, but not all of them. The official index covers SEEK, CareerOne and Workforce Australia, which is also not every ad.
  • How long an ad stays open: the daily search only asks for recently posted ads, so an ad drops out of it long before it closes.

Same method as the official count

The Internet Vacancy Index does, nationally, what this project does on a sample: take raw ads, remove duplicates, classify occupation and place, and count over time. It gets every ad from three job boards; this project searches Adzuna. It classifies into the official occupation codes; this project classifies by job title, and the accuracy of that is being measured by hand on a sample of 100 titles.

How the numbers are checked

Everything stored can be exported as CSV. An independent Python script, written with no extra libraries, recomputes each rule from the raw ads, every summary on this site, and each of the official index’s values against the original government spreadsheet, then compares them with what the database says.

Sources