Migrating from New Relic Drop Rules to Pipeline Cloud Rules: A Terraform Guide

[MyrinNew]

The Deprecation Notice

If you're using New Relic's newrelic_nrql_drop_rule resource in Terraform, you've likely seen the deprecation warning at the resource page: these resources will be deprecated on January 6th, 2026. This gives us still some months to migrate to the new newrelic_pipeline_cloud_rule resource.


If you want to check the NR official announcement, it's here)


In this quick article (which I wrote while multitasking preparing some burnt ends for the lunch!), I'll walk you through a real-world migration scenario, showing how we moved from the legacy drop rules to the new pipeline cloud rules while maintaining the same data filtering functionality.


Table of Contents

1. Understanding the Change
2. Our Legacy Setup
3. The New Pipeline Cloud Rules
4. Migration Strategy
5. Implementation Examples
6. Testing and Validation
7. Best Practices

Understanding the Change

Why the Change?

New Relic is consolidating their data management capabilities under the Pipeline Control umbrella. This new system offers:

• Unified data management: Both cloud rules and gateway rules in one place
• Better scalability: Designed for modern, high-volume telemetry data
• Enhanced flexibility: More granular control over data processing
• Future-proof architecture: Built to support upcoming features

Key Differences

Resource Name	newrelic_nrql_drop_rule	newrelic_pipeline_cloud_rule
NRQL Format	SELECT * FROM ... WHERE ...	DELETE FROM ... WHERE ... (Note: In UI, you write SELECT but it converts to DELETE)
Management	Standalone rules	Part of Pipeline Control
Scope	Account-level only	Account and organization-level

Our Legacy Setup

At my previous company, we had an Terraform configuration to drop data from our QA Kubernetes cluster, it was setup like this (I removed any sensitive data before publishing it just in case):






# Define Clusters and Namespaces with a Single Enable/Disable Flag
locals {
namespaces = {
analytics-qa = true,
azure-workload-identity-system = true,
cert-manager = true,
etl = true,
kafka = true,
keda = true,
keycloak = true,
kube-system = true,
newrelic = true,
rabbitmq-cluster = true,
kyverno = true,
tools = true
}

drop_rules_enabled = [
for namespace, enabled in local.namespaces : namespace if enabled
]
}

# Single Drop Rule for all types
resource "newrelic_nrql_drop_rule" "drop_data_by_namespace" {
for_each = toset(local.drop_rules_enabled)

action = "drop_data"
description = "Drop data from QA cluster in ${each.key} namespace"
nrql = "SELECT * FROM Log, CustomEvent, Span, Metric WHERE namespace_name = '${each.key}' and cluster_name = 'my-aks-cluster'"
}

# Drop additional data regardless of namespace
resource "newrelic_nrql_drop_rule" "drop_from_all_namespaces" {
action = "drop_data"
description = "Drop additional data from QA cluster, regardless of namespace"
nrql = "SELECT * FROM NginxSample, K8sPodSample, K8sNodeSample, K8sContainerSample, NFSSample, WowzaSample, K8sHpaSample, K8sJobSample, NetworkSample, ProcessSample, StorageSample, ContainerSample, K8sVolumeSample, flexStatusSample, K8sClusterSample, K8sCronjobSample, K8sServiceSample, K8sEndpointSample, KafkaBrokerSample, KafkaOffsetSample, K8sDaemonsetSample where clusterName like '%qa%' or entityName LIKE '%qa%' or env LIKE '%non-prod%' or env like '%qa%'"
}







The New Pipeline Cloud Rules

The new resource uses a different NRQL syntax and structure. Here's the equivalent configuration:






# Define Clusters and Namespaces with a Single Enable/Disable Flag
locals {
namespaces = {
analytics-qa = true,
azure-workload-identity-system = true,
cert-manager = true,
etl-qa = true,
kafka = true,
keda = true,
keycloak = true,
kube-system = true,
newrelic = true,
rabbitmq-cluster = true,
kyverno = true,
tools = true
}

drop_rules_enabled = [
for namespace, enabled in local.namespaces : namespace if enabled
]
}

# New Pipeline Cloud Rule for namespace-based filtering
resource "newrelic_pipeline_cloud_rule" "drop_data_by_namespace" {
for_each = toset(local.drop_rules_enabled)

name = "drop-qa-cluster-${each.key}-namespace"
description = "Drop data from QA cluster in ${each.key} namespace"
nrql = "DELETE FROM Log, CustomEvent, Span, Metric WHERE namespace_name = '${each.key}' AND cluster_name = 'my-aks-cluster-name'"
account_id = var.new_relic_account_id
action = "drop_data"
}

# New Pipeline Cloud Rule for infrastructure samples
resource "newrelic_pipeline_cloud_rule" "drop_from_all_namespaces" {
name = "drop-qa-cluster-infrastructure"
description = "Drop additional data from QA cluster, regardless of namespace"
nrql = "DELETE FROM NginxSample, K8sPodSample, K8sNodeSample, K8sContainerSample, NFSSample, WowzaSample, K8sHpaSample, K8sJobSample, NetworkSample, ProcessSample, StorageSample, ContainerSample, K8sVolumeSample, flexStatusSample, K8sClusterSample, K8sCronjobSample, K8sServiceSample, K8sEndpointSample, KafkaBrokerSample, KafkaOffsetSample, K8sDaemonsetSample WHERE clusterName LIKE '%qa1%' OR entityName LIKE '%qa%' OR env LIKE '%non-prod%' OR env LIKE '%qa%'"
action = "drop_data"
}







Key Changes in the Migration

1. Resource Type: Changed from newrelic_nrql_drop_rule to newrelic_pipeline_cloud_rule
2. NRQL Syntax:
   • Old: SELECT * FROM ... WHERE ...
   • New: DELETE FROM ... WHERE ...
3. Required Parameters:
   • Added name parameter (unique identifier)
   • Added account_id parameter
   • Removed action parameter (always "drop_data" in new system)
4. Naming Convention: Used kebab-case for rule names to improve readability

Migration Strategy

Step 1: Plan the Migration

Before starting the migration, ensure you have:

1. Inventory of existing rules: Document all current drop rules
2. Testing environment: Set up a test account if possible
3. Rollback plan: Keep backup of old configuration
4. Monitoring setup: Track data ingestion before and after migration

Step 2: Create Migration Configuration

Create a new Terraform configuration file alongside your existing one:






# variables.tf
variable "new_relic_account_id" {
description = "New Relic Account ID"
type = string
}

variable "enable_pipeline_rules" {
description = "Feature flag to enable new pipeline rules"
type = bool
default = false
}

# versions.tf
terraform {
required_version = ">= 1.10"
required_providers {
newrelic = {
source = "newrelic/newrelic"
version = ">= 3.74.0" # Ensure you have the latest version
}
}
}







Step 3: Implement Feature Toggle

Use a feature flag to switch between old and new rules:






# main.tf
locals {
use_pipeline_rules = var.enable_pipeline_rules
}

# Old rules (will be removed after migration)
resource "newrelic_nrql_drop_rule" "drop_data_by_namespace" {
for_each = local.use_pipeline_rules ? {} : toset(local.drop_rules_enabled)

action = "drop_data"
description = "Drop data from QA cluster in ${each.key} namespace"
nrql = "SELECT * FROM Log, CustomEvent, Span, Metric WHERE namespace_name = '${each.key}' and cluster_name = 'my-aks-cluster-name'"
}

# New rules
resource "newrelic_pipeline_cloud_rule" "drop_data_by_namespace" {
for_each = local.use_pipeline_rules ? toset(local.drop_rules_enabled) : {}

name = "drop-qa-cluster-${each.key}-namespace"
description = "Drop data from QA cluster in ${each.key} namespace"
nrql = "DELETE FROM Log, CustomEvent, Span, Metric WHERE namespace_name = '${each.key}' AND cluster_name = 'my-aks-cluster-name'"
account_id = var.new_relic_account_id
action = "drop_data"
}







Implementation Examples

Example 1: Drop Attributes from Specific Data Types

If you need to drop specific attributes instead of entire events and on specific environments (in this example, from production):






resource "newrelic_pipeline_cloud_rule" "drop_sensitive_attributes" {
name = "drop-sensitive-data"
description = "Remove PII from logs in production"
nrql = "DELETE email, ssn, credit_card FROM Log WHERE environment = 'production'"
account_id = var.new_relic_account_id
action = "drop_data"
}







Example 2: Complex Filtering Rules

For more complex scenarios with multiple conditions:






resource "newrelic_pipeline_cloud_rule" "complex_drop_rule" {
name = "drop-complex-conditions"
description = "Drop data based on multiple conditions"
nrql = EOT
DELETE FROM Metric, Span
WHERE (service_name IN ('test-service', 'debug-service')
AND environment = 'development')
OR (error_rate > 0.95 AND environment = 'qa')
EOT
account_id = var.new_relic_account_id
action = "drop_data"
}







Example 3: Using Dynamic Blocks for Multiple Rules





locals {
drop_rules = {
"drop-test-logs" = {
description = "Drop test environment logs"
nrql = "DELETE FROM Log WHERE environment = 'test'"
}
"drop-debug-metrics" = {
description = "Drop debug metrics"
nrql = "DELETE FROM Metric WHERE debug = true"
}
"drop-staging-traces" = {
description = "Drop staging traces older than 1 hour"
nrql = "DELETE FROM Span WHERE environment = 'staging' AND timestamp
}
}
}

resource "newrelic_pipeline_cloud_rule" "dynamic_rules" {
for_each = local.drop_rules

name = each.key
description = each.value.description
nrql = each.value.nrql
account_id = var.new_relic_account_id
}







Testing and Validation

Verify Rules Are Working

After creating pipeline cloud rules, validate they're working correctly:






-- Check if data is being dropped (should see count drop to 0)
SELECT count(*) FROM Log
WHERE namespace_name = 'analytics-qa'
AND cluster_name = 'my-aks-cluster-name'
TIMESERIES SINCE 30 minutes ago

-- Verify other data is not affected
SELECT count(*) FROM Log
WHERE namespace_name != 'analytics-qa'
TIMESERIES SINCE 30 minutes ago







Monitor Data Ingestion

Create a dashboard to monitor the impact:






resource "newrelic_one_dashboard" "pipeline_rules_monitor" {
name = "Pipeline Rules Monitoring"
description = "Monitor the impact of pipeline cloud rules"

page {
name = "Data Ingestion Overview"

widget_line {
title = "Data Ingestion by Namespace"
row = 1
column = 1
width = 6
height = 3

nrql_query {
query = EOT
SELECT count(*)
FROM Log, Metric, Span
FACET namespace_name
TIMESERIES AUTO
EOT
}
}

widget_billboard {
title = "Total Data Points Dropped"
row = 1
column = 7
width = 3
height = 3

nrql_query {
query = EOT
SELECT filter(count(*), WHERE namespace_name IN (${join(",", [for ns in keys(local.namespaces) : "'${ns}'"])}))
FROM Log, Metric, Span
SINCE 1 hour ago
EOT
}
}
}
}







Best Practices

1. Use Descriptive Names

Pipeline cloud rules require unique names. Use a consistent naming convention:






# Good naming examples
name = "drop-${environment}-${data_type}-${condition}"
name = "drop-qa-logs-by-namespace"
name = "remove-pii-from-production-logs"

# Avoid generic names
name = "rule1" # Bad
name = "drop-data" # Too generic







2. Document Your Rules

Always include meaningful descriptions:






resource "newrelic_pipeline_cloud_rule" "example" {
name = "drop-high-volume-debug-logs"
description = "Reduces data ingestion costs by dropping verbose debug logs from non-production environments. This rule saves approximately 500GB/day."
nrql = "DELETE FROM Log WHERE log_level = 'DEBUG' AND environment != 'production'"
account_id = var.new_relic_account_id
}







3. Test in Non-Production First

Always test your rules in a non-production environment:






# Use environment-specific configurations
resource "newrelic_pipeline_cloud_rule" "test_rule" {
count = var.environment == "test" ? 1 : 0

name = "test-drop-rule-${var.environment}"
description = "Testing pipeline rule in ${var.environment}"
nrql = "DELETE FROM Log WHERE test_flag = true"
account_id = var.new_relic_account_id
}







4. Monitor Rule Performance

Track the impact of your rules on data ingestion:






-- Query to monitor data dropped by rules
SELECT bytecountestimate()
FROM Log, Metric, Span
WHERE namespace_name IN ('analytics-qa', 'etl-qa', 'tools')
FACET namespace_name
SINCE 1 day ago
COMPARE WITH 1 day ago







5. Gradual Migration

Migrate rules gradually to minimize risk:






# Phase 1: Migrate low-impact rules
# Phase 2: Migrate medium-impact rules
# Phase 3: Migrate critical rules

locals {
migration_phase = 1

rules_by_phase = {
1 = ["test", "development"]
2 = ["staging", "qa"]
3 = ["production"]
}

active_environments = flatten([
for phase, envs in local.rules_by_phase :
envs if phase local.migration_phase
])
}







Common Pitfalls and Solutions

Pitfall 1: NRQL Syntax Differences

Problem: Confusion between SELECT and DELETE syntax






# Wrong - using old drop rule syntax
nrql = "SELECT * FROM Log WHERE environment = 'test'"

# Correct - pipeline cloud rules use DELETE
nrql = "DELETE FROM Log WHERE environment = 'test'"







Note: If you're testing in the New Relic UI, you'll write SELECT queries which are automatically converted to DELETE when saved. However, in Terraform, you must use DELETE syntax directly.


Pitfall 2: Case Sensitivity

Problem: NRQL keywords must be uppercase






# Wrong
nrql = "delete from Log where environment = 'test'"

# Correct
nrql = "DELETE FROM Log WHERE environment = 'test'"







Pitfall 3: Missing Account ID

Problem: Forgetting to specify account_id






# This will fail
resource "newrelic_pipeline_cloud_rule" "example" {
name = "drop-test-logs"
description = "Drop test logs"
action = "drop_data"
nrql = "DELETE FROM Log WHERE environment = 'test'"
# Missing: account_id = var.new_relic_account_id
}







Rollback Plan

If issues arise during migration, have a rollback plan ready:






# Keep old rules in a separate file: legacy_drop_rules.tf.backup
# Quick rollback procedure:
# 1. Set enable_pipeline_rules = false
# 2. Run terraform apply
# 3. Verify old rules are active
# 4. Monitor data ingestion

# Emergency rollback script
#!/bin/bash
terraform apply -var="enable_pipeline_rules=false" -auto-approve
echo "Rollback completed. Monitoring data ingestion..."







Timeline for Migration

Here's a suggested timeline for your migration:

1. Now - Q1 2025: Test new pipeline cloud rules in development
2. Q2 2025: Begin migration in staging environments
3. Q3 2025: Migrate production rules with careful monitoring
4. Q4 2025: Complete migration and remove old resources
5. Before Jan 6, 2026: Final verification and cleanup

Conclusion

Migrating from newrelic_nrql_drop_rule to newrelic_pipeline_cloud_rule is straightforward but requires careful planning. The new Pipeline Cloud Rules offer better integration with New Relic's evolving data management platform and will ensure your infrastructure remains compatible with future updates.


Key takeaways:

• Start migration early to allow time for testing
• Use feature flags for gradual rollout
• Monitor the impact on your data ingestion
• Keep detailed documentation of your rules
• Have a rollback plan ready

Resources

• New Relic Pipeline Control Documentation
• Terraform Provider Documentation
• Drop Rules EOL Announcement
• New Relic Community Forums

Have you started your migration yet? Share your experiences and tips in the comments below!


newrelic #terraform #devops #observability #monitoring #infrastructure #cloudnative #kubernetes



More...