Data Methodology

1. Data Collection

Job postings are collected from APIs from three job sources: Greenhouse, Ashby, and Lever. Jobs are not filtered by any keyword until they're collected.

• Coverage: We map companies to their ATS boards and crawl them on a schedule. Each full crawl processes roughly 24,000 raw postings, of which about 4,600 match our cloud/devops relevance criteria. Over time, we add more companies to the search as they post relevant job listings.
• Freshness: A daily freshness check re-crawls every board. A job present in its board's API response is verified live; when an employer removes a posting, it disappears from the board and is deleted from our database within 24 hours.
• Relevance: We keep jobs that mention tracked cloud technologies and either match a cloud/devops role taxonomy (e.g., SRE, platform engineer, solutions architect) or whose descriptions are technical enough (3+ tracked technologies). We do our best to exclude non-engineering titles.

2. Salary Extraction

Salaries come only from postings that state them: structured compensation fields where the ATS provides them, otherwise explicit salary ranges or hourly rates found in the description text. There is no statistical modeling used to estimate salaries. Thus, some jobs won't have salaries.

• Normalization: Ranges are converted to annual figures (hourly wages are annualized at 2,080 hours/year), and implausible ranges (generally superlow outliers) are discarded.
• Note on Averages: When we report a "Median Salary", we take the midpoint of each stated salary band and calculate the median across all matching jobs. We also report the 25th and 75th percentiles so you can see the spread; job data generally follows a normal distribution.

3. Skill and Certification Tracking

To determine which skills are "trending", we parse the raw text of job descriptions.

• Keyword Matching: We match job text against a shared taxonomy of tracked technologies (e.g., "Kubernetes", "EKS", "Terraform") using word-boundary matching to avoid false positives.
• Certifications: We track certification codes, full names, and common aliases (e.g., "AWS Solutions Architect Professional", "AWS SAP") to determine the demand for specific certs.

4. Quality Control

Because our data comes from the employer's own recruiting system, a listing in our database is verified live by definition — "ghost jobs" are removed within a day of the employer taking them down, not after lingering for weeks. Beyond that:

• Board validation: Company-to-board mappings are periodically re-validated against the live APIs; dead or empty boards are flagged, and mappings are rejected when a board's content no longer actually belongs to the company.
• Health checks: Each crawl is monitored — it fails loudly if more than 20% of boards error out or if the verified job corpus drops below our quality floor, so silent data loss gets caught.
• Known gaps: Some large enterprises host their job boards on ATS platforms without a public API (e.g., Workday); those employers aren't covered yet.

Contact

Questions about our data? Feel free to reach out to john@cloudjobs.io.