From Jupyter Notebooks to Scheduled Data Pipelines
This isn't just about laziness. Manual execution is a reproducibility risk: - What if you change something mid-week? - What about version control? - How do you track execution history?
Notice how I truncated tweet text and rounded numbers. This isn't just formatting — it's reducing payload size, which matters if you're delivering via chat apps or email.
Discord webhooks accept Markdown, but don't render Pandas DataFrames. This is why I converted the report into a simple message structure. For richer formatting, consider: - Sending JSON as file attachment - Using embeds (discord.js-style payloads) - Generating PNGs of plots with matplotlib
Cron uses localhost timezone. To debug timing:\nbash date timedatectl\n Ensure your system timezone matches expectation.
This isn't just paranoia. I once had a cron job fail silently because my machine swapped to death during backup procedures. System checks catch that.
If you're considering cloud schedulers, compare: - Cron: $0, unlimited runs - GitHub Actions: $0 for private repos (200 runs/month) - AWS Lambda: $0.20/1M requests + $0.0000002 per GB-second - Airflow/ECS: $7+ per month minimum
Rate limiting isn't academic. With free APIs, production usage often violates assumptions made during notebook development.
Even when moving to Airflow/Dagster, the core business logic (`report.py`) remains identical. The scheduling wrapper is the only thing swapped out.