# Module 07: Synthetic Monitoring Basics (Labs 10-13) Continuous Uptime Monitoring with Grafana SM <small>Navigate: [All Slides](../index.html) | [Prev: Cloud Integration](../06_Cloud_Integration/index.html) | [Next: Browser Testing](../08_Browser_Testing/index.html)</small> <aside class="notes"> Labs 10 through 13: Grafana Synthetic Monitoring introduction, HTTP checks, DNS/TCP checks, and scripted checks. </aside>

## What is Synthetic Monitoring? Continuous, scheduled checks from global probe locations that answer: **"Is my service reachable and working correctly right now?"** From the perspective of users around the world. <aside class="notes"> Synthetic Monitoring is fundamentally different from load testing. Synthetic Monitoring runs 1 VU on a schedule to verify functionality, while load testing runs many VUs to find performance limits. </aside>

## SM vs Load Testing | | Grafana SM | k6 Load Testing | |---|---|---| | Question | Is it up? | How does it perform under load? | | Trigger | Scheduled (every N minutes) | On-demand (manual or CI) | | Virtual users | 1 | Many | | Duration | Seconds | Minutes to hours | | Goal | Detect outages | Find capacity limits | You use both: SM for always-on visibility, k6 for periodic performance analysis. <aside class="notes"> Think of Synthetic Monitoring as your smoke detector and k6 as your stress test lab. Both are essential for different purposes. </aside>

## DataDog Comparison Grafana Synthetic Monitoring = DataDog Synthetic API Tests Both run scheduled HTTP checks from global locations, both alert on failures, both display uptime percentages. **Key difference:** SM results live natively in Grafana alongside metrics, logs, and traces. No separate portal. <aside class="notes"> With DataDog, you switch between Synthetics portal and APM dashboards. With Grafana everything is in one unified interface. </aside>

## Lab 10: SM Introduction In your Grafana Cloud stack, expand **Testing & synthetics → Synthetics** in the sidebar. SM sub-nav: - **Checks** — main list of all your monitors - **Probes** — global probe locations (AMER / APAC / EMEA) - **Alerts (Legacy)** — pre-built alerting rules; new work lives under **Alerts & IRM** - **Config** — plugin settings The fleet-level view lives on the **Synthetics** landing page itself. <aside class="notes"> The Checks list is your operational view. The landing page plus check-detail dashboards replaced the old Summary tab. Alerts (Legacy) is retained for backwards compatibility; new alert rules should use the unified Alerts & IRM section. </aside>

## Creating Your First HTTP Check **Checks → + Create new check → API Endpoint** 5-step wizard: **Request → Uptime → Labels → Execution → Alerting** ```yaml Request Job name: Workshop Demo Request type: HTTP (radio; row also has Ping/DNS/TCP/Traceroute) Request target: https://grafana.com Uptime Valid status codes: 2xx (default) Execution Probe locations: North Virginia US, Frankfurt DE, Singapore SG Frequency: 1m pill ``` Use the right-hand **Test** panel for immediate execution without waiting. <aside class="notes"> The Test button is invaluable during setup — instant feedback without polluting uptime history. The old single-form check-creation screen has been replaced by a 5-step wizard. </aside>

## Understanding Check Results Check detail page shows: - **Uptime percentage** (100% after successful runs) - **Response time graph** by probe location - **Probe breakdown table** with per-location stats - **Error log** for failed attempts Geographic latency differences are visible: US East will be faster to grafana.com than Asia Pacific. <aside class="notes"> This view directly mirrors what users in those regions experience. If AP users complain it's slow, this graph is the first place to look. </aside>

## Lab 11: HTTP Validation Beyond status code 200, add: - **Regexp validation** — assert body or header content with a regex (replaces old "Body Contains" / "Body Matches Regex" assertions) - **Valid status codes** range — 2xx by default - **TLS Validation** — detect certificate expiry (on by default) ```yaml Uptime step Valid status codes: 2xx Regexp validation: Source: Body Match condition: slideshow Invert: ✓ (fail unless body matches) Request → Request options → TLS Disable target certificate validation: unchecked (TLS validation on) ``` > Response-time pass/fail is gone — latency is a global threshold now, not a per-check assertion. <aside class="notes"> Certificate expiry is one of the most common causes of unexpected outages. Synthetic Monitoring with TLS validation catches this weeks before expiry. The old typed assertions (Body Contains, Body Matches Regex, Header Contains) are unified into a single Regexp validation model — default semantics is "fail when regex matches", so "must contain" checks need the Invert box. </aside>

## Custom Request Headers ```yaml Header name: X-Workshop Header value: true ``` Use cases: - Bypass WAF rules with secret headers - Activate feature flags - Mark traffic in logs (filter synthetic requests from analytics) <aside class="notes"> On your own services you might send X-Synthetic-Monitor: grafana-sm and filter those requests from error rate SLI. </aside>

## Check Status Logic A check is **Up** when ALL of the following pass: - HTTP connection established within timeout - TLS validation passes (if enabled) - HTTP response received - All assertions pass A check is **Down** when ANY fail: - Connection timeout or refused - TLS certificate invalid/expired - Any assertion fails <aside class="notes"> Uptime = successful runs / total runs. This calculation is per-probe and aggregate, giving both global and regional views. </aside>

## Lab 12: DNS and TCP Checks **DNS Check** — verify hostname resolves to expected records ```yaml Target: grafana.com Record type: A Assertion: Answer contains grafana.com ``` **TCP Check** — verify port is open and accepting connections ```yaml Target: grafana.com:443 TLS: Enabled ``` <aside class="notes"> DNS checks detect DNS hijacking or propagation issues after CDN migrations. TCP checks work for databases, message brokers — anything non-HTTP. </aside>

## When to Use Each Check Type | Scenario | Check Type | |---|---| | Web pages and REST APIs | HTTP | | Multi-step user flow | Scripted (Lab 13) | | Hostname resolution | DNS | | DNS hijacking detection | DNS with assertions | | Non-HTTP services (databases, Redis, Kafka) | TCP | | TLS cert validity on non-HTTP port | TCP with TLS | <aside class="notes"> A complete monitoring strategy uses all three together: HTTP for public endpoint, DNS for hostname, TCP for backend services. </aside>

## DNS Check Results Results show: - **Reachability** — percentage that resolved - **Resolution time** — pure DNS lookup latency (5-50ms typical) - **Response by probe** — table per location Resolution time varies by location: Singapore probe querying US nameserver will be slower than Virginia. <aside class="notes"> DNS check response time is NOT HTTP latency. Comparing the two directly is not useful — they measure different layers. </aside>

## TCP Check Results For TCP checks with TLS: - **Connection time** — TCP three-way handshake (SYN, SYN-ACK, ACK) - **TLS handshake time** — time to negotiate TLS session - **Total duration** — TCP + TLS combined Typical for grafana.com:443: - TCP: 20-150ms - TLS: 30-200ms - Total: < 500ms <aside class="notes"> Unusually high connection times from a specific probe indicate network-level issues (routing, packet loss) rather than application problems. </aside>

## Lab 13: Scripted Checks Upload a k6 script to run as a scheduled synthetic monitor. **Key insight:** The scripting language is exactly the same k6 JavaScript you've been writing. No separate DSL. Constraint: SM scripted checks run with 1 VU on a schedule — not load tests. <aside class="notes"> A script you develop locally with k6 run can be uploaded to Synthetic Monitoring with minimal or no changes. </aside>

## Multi-Step Workflow Example ```javascript export default function () { // Step A: Verify headers endpoint const headersRes = http.get('https://httpbin.org/headers'); check(headersRes, { 'headers: status is 200': (r) => r.status === 200, }); sleep(1); // Step B: POST and verify echo const payload = JSON.stringify({ workflow: 'lab-13' }); const postRes = http.post('https://httpbin.org/post', payload); check(postRes, { 'post: body echoes payload': (r) => { return JSON.parse(r.body).json.workflow === 'lab-13'; }, }); } ``` <aside class="notes"> Each check name appears as an individual pass/fail row in the Synthetic Monitoring results dashboard. Choose names that make failures self-explanatory. </aside>

## Testing Scripts Locally First Always validate before uploading: ```bash k6 run scripts/starters/lab-13-starter.js ``` Watch for check pass/fail counts. Fix failures locally — diagnosing through SM UI is slower. SM ignores `vus` and `duration` options. Keep them for local testing. <aside class="notes"> A script that fails locally will fail in Synthetic Monitoring too. Local terminal output is more informative than the Synthetic Monitoring UI for debugging. </aside>

## Uploading to SM 1. Testing & synthetics → Synthetics → Checks → **+ Create new check** → **Scripted** 2. Paste full script content 3. Configure: - Job name - Frequency (e.g., 5 minutes) - Timeout (60s for multi-step scripts) - Probe locations (2-3 locations sufficient) Use **Test** button to verify script runs in SM environment. <aside class="notes"> 60-second timeout is important for multi-step scripts. Default 10s is too short for meaningful workflows. </aside>

## Scripted Check Best Practices Production-quality patterns: - **Groups** — `group('step-name', () => {...})` organizes output - **Custom Trend metrics** — track per-step latency independently - **try/catch/finally** — graceful error handling - **Descriptive check names** — `'step-name: what we expected'` Keep scripts under 30 seconds execution time. <aside class="notes"> If your workflow has more than 5-6 steps, consider splitting into multiple scripted checks — easier to debug when they fail. </aside>

## DataDog Comparison: Scripted Checks | DataDog | Grafana SM | |---|---| | Multistep API Test | Scripted Check | | JSON test definition or UI builder | k6 JavaScript | | Proprietary assertion syntax | Standard k6 `check()` calls | | Separate from load test tooling | Same language as k6 load tests | Significant advantage: shared k6 skills across load testing and synthetic monitoring. <aside class="notes"> In DataDog, your load testing tool (JMeter, Gatling) and synthetic tool are separate products. In Grafana ecosystem, k6 is the language for both. </aside>

<!-- .slide: class="dense-content" --> ## Key Takeaways - SM runs 1-VU scheduled checks from global probes; load testing runs many VUs to find limits - HTTP checks validate endpoints; DNS checks verify resolution; TCP checks test connectivity - Each probe location runs independently — response time graph shows geographic latency variation - Scripted checks use the exact same k6 JavaScript as local load tests - The **Test** button provides instant feedback during check configuration - SM results are native Grafana data: dashboards, alerting, SLOs work without export <aside class="notes"> Synthetic Monitoring is the always-on layer; k6 is the on-demand layer. Together they give complete observability of availability and performance. </aside>

# Lab Complete! Ready to explore browser testing <small>Navigate: [All Slides](../index.html) | [Prev: Cloud Integration](../06_Cloud_Integration/index.html) | [Next: Browser Testing](../08_Browser_Testing/index.html)</small> <aside class="notes"> Great work! You've set up Synthetic Monitoring checks for HTTP, DNS, and TCP. Next up: browser testing with real Chromium. </aside>