Incident Summary

During a production demo, users were unable to sign up or sign in to the application. Firebase Authentication returned a 503 Service Unavailable error when attempting to create new accounts.

Diagnostic Steps

1. Check Firebase Functions Logs

firebase functions:log

This revealed repeated errors from the blocking function with audience claim mismatches.

2. Get Detailed Cloud Functions Logs via gcloud

For 1st gen functions:

gcloud functions logs read --limit=50 --project=PROJECT_ID

For 2nd gen functions (Cloud Run):

gcloud run services logs read FUNCTION_NAME --project=PROJECT_ID --region=REGION

This showed successful executions from previous days, confirming the function code itself was working.

3. List All Deployed Functions

gcloud functions list --project=PROJECT_ID

Check the RUNTIME and TRIGGER columns to verify function generation.

4. Verify Deployment Configuration

firebase deploy --only functions --dry-run

Output confirmed the function was configured as 2nd Gen but revealed an outdated firebase-functions package warning.

5. Check Package Versions

cat functions/package.json | grep firebase

Verified firebase-functions version (should be v4.0+ for proper v2 support).

6. Check Identity Platform Blocking Function Registration

Navigate to Firebase Console > Authentication > Settings > Blocking functions to verify the registered URL format matches the expected generation.

Error Signature

The Firebase Functions logs revealed the following error:

FirebaseAuthError: Firebase Auth Blocking token has incorrect "aud" (audience) claim.
Expected "run.app" but got "https://us-west1-PROJECT_ID.cloudfunctions.net/onUserCreated"

Root Cause

The application uses a beforeUserCreated blocking function to:

The function was correctly written using Firebase Functions v2 SDK:

const { beforeUserCreated } = require("firebase-functions/v2/identity");

exports.onUserCreated = beforeUserCreated(async (event) => {
  // User creation logic
});

However, Firebase Identity Platform had registered the blocking function URL as a 1st generation Cloud Functions endpoint (cloudfunctions.net) instead of the 2nd generation Cloud Run endpoint (run.app).

When Identity Platform invokes a blocking function, it generates a JWT token with an aud (audience) claim. The Firebase Admin SDK validates this claim against the expected Cloud Run URL format. The mismatch between the registered URL and the expected format caused the validation to fail, resulting in the 503 error.

Resolution

  1. Redeployed the blocking function to force re-registration:

    firebase deploy --only functions:onUserCreated
    
  2. Verified the deployment showed “2nd Gen” in the output:

    functions[onUserCreated(us-west1)] Successful update operation.
    
  3. If the issue persists after redeployment, manually update the blocking function registration:

    • Navigate to Firebase Console > Authentication > Settings > Blocking functions
    • Remove the existing registration
    • The function should auto-register with the correct Cloud Run URL on next deployment

Prevention

Technical Details

ComponentExpectedActual (Broken)
Function Generation2nd Gen (Cloud Run)1st Gen (Cloud Functions)
URL Pattern*.run.app*.cloudfunctions.net
Token AudienceCloud Run URLCloud Functions URL

The Firebase Admin SDK’s FirebaseTokenVerifier performs strict audience validation. When the Identity Platform sends a token with a cloudfunctions.net audience to a function expecting run.app, the verification fails with auth/argument-error.