Slack Overrides

Override your NightOps schedules from Slack when you need resources outside their normal operating hours.

When to Use Overrides

Common scenarios:

• Working late on a deadline
• Weekend deployment or testing
• Debugging a production issue
• Demo for a customer in different timezone
• On-call incident investigation

---

Starting a Stopped Collection

When a collection has been stopped by its schedule, start it manually:

/nightops start staging-api

Response:

⚡ Starting "Staging API" (5 resources)

Note: This collection is normally off at this time.
Schedule: Weekday Business Hours (8am-8pm ET)

Starting resources...
✓ api-server-1     Started
✓ api-server-2     Started
✓ worker-1         Started
✓ rds-primary      Started (may take 5-10 min)
✓ cache-cluster    Started

Staging API is now running.

⏰ Auto-stop: This collection will stop at the next scheduled time (Monday 8:00 PM ET)
   Or run `/nightops stop staging-api` when done.

---

Extending an Active Schedule

Keep resources running beyond their scheduled stop time:

/nightops extend staging-api 2h

Response:

⏰ Extended "Staging API"

Original stop time: Today 8:00 PM
New stop time: Today 10:00 PM

Resources will continue running for 2 more hours.

Extension Options

/nightops extend staging-api 2h      # 2 hours
/nightops extend staging-api 30m     # 30 minutes
/nightops extend staging-api 1d      # 1 day (until same time tomorrow)
/nightops extend staging-api friday  # Until Friday at scheduled stop time

---

Override Options

Keep On Until Next Schedule

Resources stay on until the next scheduled start time:

/nightops start staging-api --until-schedule

Example: If it's Saturday and schedule is Mon-Fri 8am-8pm:

• Resources start now
• Stay on until Monday 8pm (first scheduled stop)

Keep On For Duration

Specify how long resources should run:

/nightops start staging-api --for 4h

Resources will automatically stop after 4 hours.

Keep On Until Time

Specify an exact stop time:

/nightops start staging-api --until "10pm"
/nightops start staging-api --until "2024-01-15 18:00"

---

Stopping an Override

When you're done early, stop the resources:

/nightops stop staging-api

Response:

🔴 Stopping "Staging API" (5 resources)

Stopping resources...
✓ api-server-1     Stopped
✓ api-server-2     Stopped
✓ worker-1         Stopped
✓ rds-primary      Stopped
✓ cache-cluster    Stopped

Staging API is now stopped.

Note: Regular schedule resumes. Next start: Monday 8:00 AM ET

---

Requesting Overrides

For collections with approval required:

/nightops start production-api

Response:

🔔 Approval Required

You're requesting to start "Production API" outside its schedule.

Please provide a reason:

After entering reason:

📤 Request Submitted

Your request to start "Production API" has been sent to #infrastructure-approvers.

Reason: "Debugging customer issue #4521"

You'll be notified when approved or denied.

---

Override Notifications

When someone starts an override, the team is notified:

#staging-ops

⚡ Override Alert

Alice started "Staging API" outside scheduled hours.

Reason: "Weekend deployment testing"
Started: Saturday 2:30 PM
Auto-stop: Saturday 6:30 PM (4 hours)

[View in Dashboard]

---

Weekend Overrides

Start for Weekend Work

/nightops start staging-api --for 4h -r "Weekend feature testing"

Start for Full Weekend

/nightops start staging-api --until monday

Resources will run until Monday's scheduled start time.

---

Scheduled Overrides

Plan overrides in advance:

Schedule a Future Start

/nightops schedule-override staging-api start "Saturday 9am" --for 6h

Response:

📅 Override Scheduled

"Staging API" will start:
  Start: Saturday 9:00 AM
  Stop: Saturday 3:00 PM (6 hours)

Reason: (none provided)
Cancel with: /nightops cancel-override staging-api

View Upcoming Overrides

/nightops overrides

Response:

📅 Upcoming Overrides

staging-api
  Start: Saturday 9:00 AM
  Stop: Saturday 3:00 PM
  By: alice@company.com

qa-environment
  Extended until: Friday 10:00 PM
  By: bob@company.com

[Cancel Override...]

Cancel Planned Override

/nightops cancel-override staging-api

---

Override History

View recent overrides:

/nightops history staging-api

Response:

📜 Recent Activity: Staging API

Today
  2:30 PM  ⚡ Started (override) by alice@company.com
           Reason: "Weekend deployment"
  6:30 PM  🔴 Auto-stopped (override ended)

Yesterday
  8:00 AM  🟢 Scheduled start
  9:45 PM  ⏰ Extended 2h by bob@company.com
  11:45 PM 🔴 Auto-stopped (extension ended)

Wednesday
  8:00 AM  🟢 Scheduled start
  8:00 PM  🔴 Scheduled stop

---

Quick Commands

Status with Override Info

/nightops status staging-api

Response (when override active):

⚡ Staging API: Running (Override)

Status: Running
Override: Active until 6:30 PM (2h 15m remaining)
Started by: alice@company.com
Reason: "Weekend deployment"

Resources:
  🟢 api-server-1     Running
  🟢 api-server-2     Running
  🟢 worker-1         Running
  🟢 rds-primary      Running
  🟢 cache-cluster    Running

[Stop Now] [Extend] [View Details]

---

Override Policies

Configuring Override Limits

Admins can set override policies:

1. Go to Settings → Policies → Overrides
2. Configure limits:

Setting	Description	Example
Max duration	Maximum override length	8 hours
Require reason	Require reason for overrides	Yes
Notify channel	Channel for override notifications	#staging-ops
Require approval	Collections requiring approval	production-*
Weekend limits	Different limits for weekends	4 hours max

Per-Collection Settings

Override individual collection policies:

1. Edit the collection
2. Under Override Policy, configure:
   • Max override duration
   • Approval requirements
   • Auto-stop behavior

---

Best Practices

Always Provide Reasons

Helps with auditing and team awareness:

/nightops start staging-api -r "Testing PR #123 deployment"

Set Auto-Stop

Prevent forgotten resources from running all weekend:

/nightops start staging-api --for 4h

Notify the Team

Use a channel notification:

/nightops start staging-api -r "Load testing" --notify #backend-team

Clean Up When Done

Don't wait for auto-stop if you're done early:

/nightops stop staging-api

---

Override Costs

Overrides add to your monthly costs. NightOps tracks override impact:

View Override Cost Impact

/nightops cost-impact staging-api

Response:

💰 Cost Impact: Staging API

This Month:
  Scheduled hours: 260 hrs
  Override hours: 18 hrs
  Total: 278 hrs

Override Cost: $61.74 (additional)

Override Breakdown:
  Sat Jan 13: 6 hrs ($20.58)
  Wed Jan 10: 4 hrs ($13.72)
  Tue Jan 9: 8 hrs ($27.44)

Monthly Override Summary

/nightops cost-impact --month

---

Troubleshooting

Override Not Starting

1. Check you have permission to start the collection
2. Verify the collection exists: /nightops list
3. Check if approval is required

Resources Not Coming Up

Some resources take time to start:

• RDS: 5-10 minutes
• EKS node groups: 3-5 minutes
• Large EC2 instances: 1-2 minutes

Override Stopped Unexpectedly

Check if:

• Auto-stop time was reached
• Someone else stopped the collection
• A scheduled stop occurred

View history:

/nightops history staging-api

Can't Extend Override

1. Check max duration policy
2. Verify you have permission
3. Try a shorter extension

---

Next Steps

• View schedule examples — Optimize your schedules
• Configure Slack integration — Full Slack setup
• Set up calendar — Track overrides in calendar