Custom Email Notifications for Databricks Pipeline Failures

When working with Databricks pipelines and workflows, failures are inevitable. While Databricks provides built-in notifications for job failures, these alerts are often not customizable and may not fit specific reporting or formatting needs. A more flexible and cost-effective approach is to set up custom email notifications that include pipeline details and error messages in a structured format, such as an Excel attachment.

This blog walks through approaches to implement custom notifications using SMTP, SendGrid, Azure Logic Apps, and Microsoft Graph API.

Why Custom Notifications?

• Flexible formatting: Include pipeline metadata, error messages, and runtime details.
• Attachments: Share structured reports (Excel, CSV, etc.) instead of plain text.
• Cost efficiency: Avoid additional third-party monitoring solutions.
• Integration options: Easily plug into existing email infrastructure.

Approach 1: SMTP-Based Notifications

Using SMTP with Python inside a Databricks notebook, you can generate an Excel report and send it via email whenever a pipeline fails.

Example Implementation

import smtplib from email.mime.multipart import MIMEMultipart from email.mime.base import MIMEBase from email.mime.text import MIMEText from email import encoders from io import BytesIO import pandas as pd  #Sample pipeline history df = spark.createDataFrame([ ('pipeline1', 'success', '7min'), ('pipeline1', 'fail', '3min'), ('pipeline1', 'success', '10min') ], ["PipelineName", "Status", "Duration"])  # Convert DataFrame to Excel output = BytesIO()  with pd.ExcelWriter(output, engine='xlsxwriter') as writer:  df_pd = df.toPandas()  df_pd.to_excel(writer, index=False, sheet_name='Sheet1')  workbook = writer.book  worksheet = writer.sheets['Sheet1'] 

# Apply formatting header_format = workbook.add_format({     'bold': True,     'bg_color': '#FFF00',     'border': 1,     'align': 'center',     'valign': 'vcenter' }) for col_num, value in enumerate(df_pd.columns):     worksheet.write(0, col_num, value.upper(), header_format)  cell_format = workbook.add_format({'border': 1}) for row in range(1, len(df_pd) + 1):     for col in range(len(df_pd.columns)):         worksheet.write(row, col, df_pd.iloc[row-1, col], cell_format)  for i, col in enumerate(df_pd.columns):     worksheet.set_column(i, i, 20) output.seek(0)  # Email configuration sender = "[email protected]" receiver = "[email protected]" subject = "Pipeline Execution Report" body = """Hello Team,  Please find the attachment of the latest pipeline report.  Thanks, Pipeline Team"""  msg = MIMEMultipart() msg['From'] = sender msg['To'] = receiver msg['Subject'] = subject msg.attach(MIMEText(body, 'plain'))  part = MIMEBase('application', 'vnd.openxmlformats-officedocument.spreadsheetml.sheet') part.set_payload(output.read()) encoders.encode_base64(part) part.add_header('Content-Disposition', 'attachment; filename="pipeline_report.xlsx"') msg.attach(part)  smtp_server = "smtp.office.com" smtp_port = 587  with smtplib.SMTP(smtp_server, smtp_port) as server:     server.starttls()     server.login(sender, "sender_password")     server.send_message(msg)  print("Email sent successfully with Excel attachment") 

Scheduling Notifications

You can automate the notification trigger by scheduling the notebook:

Option 1: Databricks Jobs

• Create or edit a Databricks job.
• Add a task dependency so the notification script runs only if the previous task fails.
• This ensures error details are captured and reported immediately.

Option 2: Azure Logic Apps

• Configure a Logic App that listens for pipeline failures.
• Pass pipeline details and attachments via an API call in JSON format.
• Logic Apps handle email delivery and retry mechanisms.

Conclusion

While Databricks provides basic failure notifications, extending them with custom SMTP or Logic App workflows ensures:

• Rich, formatted reports.
• Team visibility with detailed context.
• Seamless integration with enterprise communication tools.

This approach is cost-effective, scalable, and easily adaptable for large-scale pipeline monitoring.