A full-stack ETL and analytics platform with CSV, JSON, synthetic data, controlled web collection, manual candidate review, simplified Medallion architecture in Parquet, DuckDB and a Vue dashboard.
- FastAPI
- Vue 3
- DuckDB
- Parquet
- Pandas
- Docker
- TypeScript
Context and problem #
Remote markets change quickly and require comparable data on stack, seniority, location, salary and job source.
My contribution #
The case highlights multi-format ingestion, transparent web collection, normalization, quality metrics, lineage and an analytics product.
Architecture #
The pipeline registers sources, collects candidates, applies manual review and transforms data into Bronze, Silver and Gold layers queried by DuckDB and a Vue dashboard.
- ✓ Controlled scraper
- ✓ Manual candidate review
- ✓ Parquet layers
- ✓ DuckDB analytics
Evaluation and quality #
Quality is tracked through rules, schema drift and per-run lineage, avoiding disposable scraping workflows.
Limitations #
Collection respects public/permitted sources, robots.txt when configured, rate limits and timeouts; each source's terms still need user verification.
Next steps #
Expand connectors, observability, classification rules, trend analysis and cloud deployment.
Test the product and inspect the code
The live demo runs the core flow with safe local data. The public repository preserves the complete backend, tests, infrastructure and technical documentation.