migration_guide

Comprehensive guide for migrating Protocol Buffer schemas safely while maintaining backward/forward compatibility.

Protocol Buffer Schema Migration Guide

Comprehensive guide for migrating Protocol Buffer schemas safely while maintaining backward/forward compatibility.

Migration Strategy
Breaking vs Non-Breaking Changes
Migration Patterns
Version-by-Version Examples
Testing Migrations
Rollback Strategy
Production Checklist

Migration Strategy

Three-Phase Deployment

The safest approach for schema evolution:

Phase 1: Deploy Transitional Version (backward compatible)
  - Add new fields as optional
  - Mark old fields as deprecated
  - Support both old and new patterns

Phase 2: Migrate Data and Clients
  - Update all clients to use new fields
  - Migrate existing data
  - Monitor for deprecated field usage

Phase 3: Clean Up (potentially breaking)
  - Remove deprecated fields (reserved)
  - Drop transitional support code
  - Deploy clean version

Compatibility Modes

Backward Compatible (most common):

New servers can read old client messages
Safe for server-first deployments
Add optional fields, new enum values

Forward Compatible (less common):

Old servers can read new client messages
Safe for client-first deployments
Never remove fields

Full Compatible (ideal):

Both backward and forward compatible
Only add optional fields
Never remove anything

Breaking vs Non-Breaking Changes

✅ Non-Breaking Changes

// ✅ Adding optional fields
message User {
  string id = 1;
  string email = 2;
  string name = 3;           // New optional field
}

// ✅ Adding new enum values (at end)
enum Status {
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE = 1;
  STATUS_INACTIVE = 2;
  STATUS_ARCHIVED = 3;       // New value
}

// ✅ Marking fields deprecated
message User {
  string id = 1;
  string email = 2;
  string old_name = 3 [deprecated = true];
  string display_name = 4;   // Replacement
}

// ✅ Adding new messages/services
message UserProfile {        // New message
  string bio = 1;
}

// ✅ Reserving fields
message User {
  reserved 10, 11, 12;
  reserved "deleted_field", "removed_field";
}

❌ Breaking Changes

// ❌ Removing fields without reservation
message User {
  string id = 1;
  string email = 2;
  // string name = 3;         // REMOVED - breaks compatibility!
}

// ❌ Changing field types
message User {
  string id = 1;
  int64 user_id = 2;         // Was string, now int64 - BREAKS!
}

// ❌ Changing field numbers
message User {
  string id = 1;
  string email = 3;          // Was 2, now 3 - BREAKS!
}

// ❌ Removing enum values
enum Status {
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE = 1;
  // STATUS_INACTIVE = 2;     // REMOVED - old data becomes invalid!
}

// ❌ Changing required/optional (proto2)
message User {
  required string id = 1;    // Was optional - BREAKS!
}

// ❌ Renaming fields (without json_name)
message User {
  string id = 1;
  string user_email = 2;     // Was 'email' - JSON serialization BREAKS!
}

Migration Patterns

Pattern 1: Adding a Field

Safe, backward compatible

// Version 1
message User {
  string id = 1;
  string email = 2;
}

// Version 2 - Add optional field
message User {
  string id = 1;
  string email = 2;
  string phone = 3;          // New optional field
}

Behavior:

Old clients: Ignore phone field (unknown field)
New clients: Can read old data (phone will be empty)
No deployment coordination needed

Pattern 2: Replacing a Field

Requires three-phase deployment

// Version 1 (Original)
message User {
  string id = 1;
  string email = 2;
  string name = 3;
}

// Version 2 (Transitional - DEPLOY THIS FIRST)
message User {
  string id = 1;
  string email = 2;
  string name = 3 [deprecated = true];
  string display_name = 4;   // New field

  // Server code: Write to both, read from display_name preferring it
  // Client code: Start writing to display_name, read from both
}

// Version 3 (Final - DEPLOY AFTER ALL CLIENTS UPDATED)
message User {
  string id = 1;
  string email = 2;
  reserved 3;
  reserved "name";
  string display_name = 4;
}

Migration steps:

Deploy v2 servers (support both fields)
Migrate existing data (name → display_name)
Update all clients to use display_name
Monitor: Ensure no clients use deprecated field
Deploy v3 (remove deprecated field)

Pattern 3: Changing Field Type

Breaking change - requires new field

// Version 1
message User {
  string id = 1;
  string age = 2;            // Oops, should be int32
}

// Version 2 (Correct approach)
message User {
  string id = 1;
  string age = 2 [deprecated = true];
  int32 age_years = 3;       // New field with correct type
}

// Server migration code:
// if (user.age_years == 0 && user.age != "") {
//   user.age_years = parseInt(user.age)
// }

Never do this:

// ❌ DON'T: Reuse field number with different type
message User {
  string id = 1;
  int32 age = 2;             // Same number, different type - BREAKS!
}

Pattern 4: Enum Evolution

Safe: Add values at end

// Version 1
enum UserStatus {
  USER_STATUS_UNSPECIFIED = 0;
  USER_STATUS_ACTIVE = 1;
  USER_STATUS_INACTIVE = 2;
}

// Version 2 - Add new values
enum UserStatus {
  USER_STATUS_UNSPECIFIED = 0;
  USER_STATUS_ACTIVE = 1;
  USER_STATUS_INACTIVE = 2;
  USER_STATUS_SUSPENDED = 3;   // New value
  USER_STATUS_DELETED = 4;     // New value
}

Unsafe: Remove values

// ❌ DON'T: Remove enum values
enum UserStatus {
  USER_STATUS_UNSPECIFIED = 0;
  USER_STATUS_ACTIVE = 1;
  // USER_STATUS_INACTIVE = 2;  // Removed - old data invalid!
}

// ✅ DO: Deprecate and reserve
enum UserStatus {
  USER_STATUS_UNSPECIFIED = 0;
  USER_STATUS_ACTIVE = 1;
  reserved 2;
  reserved "USER_STATUS_INACTIVE";
}

Pattern 5: Nested Message Changes

Safe: Add optional nested message

// Version 1
message User {
  string id = 1;
  string email = 2;
}

// Version 2
message User {
  string id = 1;
  string email = 2;
  UserProfile profile = 3;   // New nested message
}

message UserProfile {
  string bio = 1;
  string avatar_url = 2;
}

Pattern 6: Service Evolution

Safe: Add new methods

// Version 1
service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
}

// Version 2
service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc ListUsers(ListUsersRequest) returns (ListUsersResponse);  // New
  rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);  // New
}

Unsafe: Change method signatures

// ❌ DON'T: Change request/response types
service UserService {
  // Was: GetUser(GetUserRequest) returns (GetUserResponse)
  rpc GetUser(GetUserRequestV2) returns (GetUserResponseV2);  // BREAKS!
}

// ✅ DO: Add new method
service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc GetUserV2(GetUserRequestV2) returns (GetUserResponseV2);  // New version
}

Version-by-Version Examples

See the example files for concrete evolution:

V1 → V2 (Non-Breaking)

user_service_v1.proto → user_service_v2.proto
Added optional fields (display_name, role, email_verified)
Added new enum value (USER_STATUS_BANNED)
Marked field deprecated (name → display_name)

V2 → V3 (Breaking with Migration)

user_service_v2.proto → user_service_v3.proto
Removed deprecated field (properly reserved)
Changed role system (UserRole → Permissions)
Added nested profile message
Required coordinated deployment

Testing Migrations

Compatibility Testing

#!/usr/bin/env python3
"""Test schema compatibility"""

import user_v1_pb2 as v1
import user_v2_pb2 as v2

# Test 1: V1 writer, V2 reader (backward compatibility)
user_v1 = v1.User(id="123", email="[email protected]", name="John")
data = user_v1.SerializeToString()

user_v2 = v2.User()
user_v2.ParseFromString(data)

assert user_v2.id == "123"
assert user_v2.email == "[email protected]"
assert user_v2.name == "John"
assert user_v2.display_name == ""  # New field is empty
print("✅ Backward compatible: V1 → V2")

# Test 2: V2 writer, V1 reader (forward compatibility)
user_v2 = v2.User(
    id="456",
    email="[email protected]",
    display_name="Jane Doe"
)
data = user_v2.SerializeToString()

user_v1 = v1.User()
user_v1.ParseFromString(data)

assert user_v1.id == "456"
assert user_v1.email == "[email protected]"
# V1 ignores unknown field (display_name)
print("✅ Forward compatible: V2 → V1")

Testing with Multiple Versions

#!/bin/bash
# Test all version combinations

# Generate code for all versions
for v in v1 v2 v3; do
  protoc --python_out=. user_service_${v}.proto
done

# Run compatibility tests
python test_v1_v2_compat.py
python test_v2_v3_compat.py
python test_v1_v3_compat.py  # Should fail if breaking changes

# Validate with tools
./scripts/analyze_schema_compatibility.py \
  --baseline user_service_v1.proto \
  --current user_service_v2.proto \
  --mode full

./scripts/analyze_schema_compatibility.py \
  --baseline user_service_v2.proto \
  --current user_service_v3.proto \
  --mode backward

Rollback Strategy

Rollback Scenarios

Scenario 1: Non-Breaking Change Rollback

V2 deployment failed → Rollback to V1
- Safe: V1 code can read V2 data (ignores new fields)
- No data migration needed

Scenario 2: Breaking Change Rollback

V3 deployment failed → Rollback to V2
- Risk: V2 code may not understand V3 data
- Need database rollback or data migration
- Always test rollback path before deployment!

Rollback Testing

#!/bin/bash
# Test rollback scenarios

# 1. Deploy V2, generate data
deploy_v2
generate_test_data_v2

# 2. Rollback to V1
rollback_v1

# 3. Verify V1 can read V2 data
test_v1_reads_v2_data || exit 1

# 4. Verify no errors in logs
check_application_logs || exit 1

echo "✅ Rollback test passed"

Production Checklist

Before deploying schema changes:

Pre-Deployment

[ ] Analyze compatibility

``bash ./scripts/analyze_schema_compatibility.py \ --baseline prod_version.proto \ --current new_version.proto \ --mode backward ``

[ ] Run validation

``bash ./scripts/validate_proto_schemas.py \ --proto-file new_version.proto ``

[ ] Generate and test code

``bash ./scripts/generate_proto_code.py \ --proto-file new_version.proto \ --language python,go \ --validate ``

[ ] Review breaking changes
Document all breaking changes
Plan migration strategy (3-phase if needed)
Communicate to all teams

[ ] Test compatibility
Write compatibility tests
Test old client → new server
Test new client → old server
Test rollback scenario

[ ] Update documentation
Update CHANGELOG
Document migration steps
Update API documentation

Deployment

[ ] Phase 1: Deploy transitional version
Supports both old and new patterns
Monitor logs for errors
Verify backward compatibility

[ ] Phase 2: Migrate data and clients
Run data migration scripts
Deploy updated clients
Monitor deprecated field usage

[ ] Phase 3: Deploy final version
Remove deprecated code
Clean up database
Update monitoring dashboards

Post-Deployment

[ ] Monitor metrics
Deserialization errors
Unknown field warnings
Application errors

[ ] Verify data integrity
Sample data checks
Field population rates
Data type correctness

[ ] Update schema registry
Register new schema version
Verify compatibility level
Update documentation

Common Pitfalls

1. Reusing Field Numbers

❌ DON'T:

message User {
  string id = 1;
  // Removed: string email = 2;
  int32 age = 2;  // Reused number 2 - DANGER!
}

Why it's bad: Old data with email="[email protected]" will be parsed as age, causing corruption.

✅ DO:

message User {
  string id = 1;
  reserved 2;
  reserved "email";
  int32 age = 3;  // New number
}

2. Changing JSON Field Names

❌ DON'T:

message User {
  string user_identifier = 1;  // Was 'id'
}

Why it's bad: JSON serialization uses field names, breaking JSON clients.

✅ DO:

message User {
  string user_identifier = 1 [json_name = "id"];  // Keep JSON name
}

3. Assuming Zero Values

❌ DON'T:

enum Status {
  STATUS_ACTIVE = 0;  // Bad: zero should be unspecified
  STATUS_INACTIVE = 1;
}

Why it's bad: Cannot distinguish between unset and explicitly set to zero.

✅ DO:

enum Status {
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE = 1;
  STATUS_INACTIVE = 2;
}

Tools and Resources

Validation Tools

# Buf: Modern protobuf tool
buf lint
buf breaking --against '.git#branch=main'

# Custom scripts
./scripts/validate_proto_schemas.py --proto-dir protos
./scripts/analyze_schema_compatibility.py --baseline v1.proto --current v2.proto

Monitoring

Set up alerts for:

Protobuf deserialization errors
Unknown field warnings
Deprecated field usage
Schema registry rejections

Documentation

Keep CHANGELOG.md updated
Document breaking changes prominently
Provide migration guides for major changes
Update API documentation automatically

Summary

Golden Rules for Safe Schema Evolution:

✅ Always add fields, never remove (use reserved)
✅ New fields must be optional
✅ Never change field numbers
✅ Never reuse field numbers
✅ Enum zero value = UNSPECIFIED
✅ Test backward compatibility
✅ Use three-phase deployment for breaking changes
✅ Monitor after deployment
✅ Have rollback plan
✅ Communicate changes to all teams

When in doubt: Add a new field rather than modifying an existing one.

migration_guide

Protocol Buffer Schema Migration Guide

Table of Contents

Migration Strategy

Three-Phase Deployment

Compatibility Modes

Breaking vs Non-Breaking Changes

✅ Non-Breaking Changes

❌ Breaking Changes

Migration Patterns

Pattern 1: Adding a Field

Pattern 2: Replacing a Field

Pattern 3: Changing Field Type

Pattern 4: Enum Evolution

Pattern 5: Nested Message Changes

Pattern 6: Service Evolution

Version-by-Version Examples

V1 → V2 (Non-Breaking)

V2 → V3 (Breaking with Migration)

Testing Migrations

Compatibility Testing

Testing with Multiple Versions

Rollback Strategy

Rollback Scenarios

Rollback Testing

Production Checklist

Pre-Deployment

Deployment

Post-Deployment

Common Pitfalls

1. Reusing Field Numbers

2. Changing JSON Field Names

3. Assuming Zero Values

Tools and Resources

Validation Tools

Monitoring

Documentation

Summary