How to Monitor JSON Payloads from Cron Jobs

Most cron monitoring tools answer one question: did the job run? That is necessary but not sufficient. A backup script can exit with code 0 while producing a corrupted file. An ETL pipeline can finish successfully while importing zero rows. A report generator can complete on time while emitting empty output. Exit codes tell you if the process crashed. They tell you nothing about the quality of the work it did.

JSON payload monitoring closes this gap. By sending a JSON object alongside your heartbeat ping, you can track business-level metrics — row counts, file sizes, durations, error rates, record counts — and get alerted when those metrics deviate from normal patterns.

Why Exit Codes Are Not Enough

Consider a daily ETL job that loads sales data into a data warehouse. On a normal day, it processes around 15,000 rows. One morning, the source API starts returning paginated results differently, and the job processes only 200 rows. The script does not crash — it handles the response correctly, writes 200 rows to the database, and exits with code 0. From every monitoring tool that only checks heartbeats or exit codes, the job is green.

The data team discovers the problem two days later when a VP asks why revenue dropped 98% in the dashboard. This scenario is not hypothetical — it happens constantly in production systems.

What JSON Monitoring Adds

When your cron job sends a JSON payload with its heartbeat, you gain three capabilities:

• Time series for every metric — Track any numeric value over time without configuring dashboards or alert rules.
• Automatic anomaly detection — Receive alerts when any metric deviates significantly from its recent average.
• Root cause context — When something goes wrong, the metrics history shows exactly what changed and when.

How Crontiq Extracts Metrics from JSON

Crontiq's Magic Engine processes JSON payloads in three steps: flattening, type detection, and storage. You do not need to register metrics, define schemas, or configure anything. Just POST JSON and Crontiq figures out the rest.

Flat Objects

The simplest case. Every numeric value at the top level becomes a metric:

String values are stored as part of the ping log but are not tracked as metrics. Only numeric types (integer, long, double) become time series data points.

Nested Objects

For complex jobs that produce structured output, Crontiq flattens nested objects using dot notation:

Each dot-separated key becomes an independent time series. Crontiq applies anomaly detection to each one individually. If source.rows_read stays stable but destination.rows_written drops, you will know the problem is in the transform or load phase, not the extraction.

Deeply Nested Objects

Crontiq handles arbitrarily deep nesting. A three-level object like {"db": {"primary": {"latency_ms": 12}}} produces the metric db.primary.latency_ms = 12. In practice, keeping your payload to two levels of nesting makes the metrics easier to read in the dashboard.

Arrays

Numeric values inside arrays are extracted using their index as part of the key:

For most use cases, named keys are preferable to arrays because the metric names are more descriptive. Instead of latencies.0, consider {"primary_latency": 12, "replica_latency": 8}.

Anomaly Detection on Metrics

Crontiq computes a rolling average over the last 10 data points for each metric key, along with the standard deviation. When a new value arrives, it checks whether the value falls within 2 standard deviations of the rolling average. If it does not, the metric is flagged as anomalous.

In mathematical terms: if |value - mean| > 2 * stddev, the value is an anomaly. This catches both sudden spikes (e.g., duration jumped from 10 seconds to 5 minutes) and sudden drops (e.g., row count went from 50,000 to 12).

When an anomaly is detected on any metric, the monitor status changes to WARNING and an alert email is sent. The alert includes the metric name, the anomalous value, the expected range, and a link to the monitor's metric history.

This approach has two important properties:

• No configuration needed — You never define thresholds, expected ranges, or alert rules. Crontiq learns what is normal from the data.
• Adapts to trends — If your row count grows 5% each week, the rolling average tracks that growth. You only get alerted on sudden deviations, not gradual changes.

Real Example: Database ETL Job

Here is a production-ready script for a database ETL job that exports order data and sends comprehensive metrics to Crontiq:

With this setup, Crontiq independently tracks four metrics. If the source database returns fewer orders than usual, if the load step drops rows, if revenue suddenly changes, or if the job takes longer than normal — each anomaly is detected and reported individually. The metric names in the alert tell you exactly which part of the pipeline is affected.

Payload Limits

Crontiq accepts POST bodies up to 100KB. The response includes a Ping-Body-Limit: 102400 header so clients can check the limit programmatically. Payloads larger than 100KB are truncated. For most monitoring use cases, the payload is a few hundred bytes — well within the limit.

You can also send plain text logs via POST. If the content type is not JSON, Crontiq stores the body as a log entry attached to the ping. This is useful for capturing script output without adding JSON formatting.