June 20, 2024
Migrating from Monolithic to Microservices: Systematic Approaches, Challenges and Recommended design patterns
Read More
Enhancing Data Processing with Kafka Connector Custom SMT
Utilizing Kafka Connectors for Seamless Integration
At our organization, we rely heavily on Kafka connectors to establish connections with external systems and applications. Leveraging these connectors, we seamlessly source and sink a significant volume of messages and data to and from Kafka. While some applications require customized data handling, they traditionally integrated these capabilities into their respective systems. However, the Event Hub Platform team has introduced an alternative solution. They have empowered application teams with the option to streamline their implementations by incorporating them into Kafka Connector Custom SMTs (Single Message Transforms). This approach offers a more centralized and efficient means of managing data customization within our Kafka ecosystem.
Enhancing Data Processing with Kafka Connector Custom SMTs
Kafka Connector Custom Single Message Transforms (SMTs) offer a versatile solution for applying tailored logic and transformations to streaming data in real-time. By leveraging Custom SMTs, organizations can streamline their data processing workflows and alleviate development burdens on application teams.
Use Cases of Custom SMTs
- Data Transformation: Convert data formats, such as Avro to JSON, to meet specific requirements.
- Message Filtering: Implement conditional filtering based on field values to selectively process messages.
- Dynamic Field Inclusion: Dynamically add new static or dynamic fields and values to data streams as per evolving needs.
- Monitoring and Reporting: Log essential identifiers (non PII) within the data stream to facilitate monitoring and reporting tasks.
- Error Handling: Route invalid messages to a Dead Letter Queue (DLQ) for further analysis and troubleshooting.
Benefits to the Application teams
- Reduced Development and Build Efforts: Implementation of the solution diminishes the workload associated with development and building processes.
- Lower Infrastructure Costs: Hosting a new microservice to meet business requirements becomes more cost-effective with reduced infrastructure costs.
- Decreased Provisioning Costs: The solution leads to decreased provisioning costs, including those related to topics, ACLs, certificates procurement and maintenance, and deployment pipelines.
- Faster Build and Deployment Times: The ability to deploy and restart the connector at any time with minimal downtime enhances efficiency in build and deployment processes, including customizations.
- Minimal Impact on Source and Target Systems: Any enhancements or deployments have minimal impact on the source and target systems, ensuring smooth operations.
Below are few of the major use cases where we have implemented custom SMT’s:
Use Case 1 – AEP Sink connector
The Digital experience domain uses the Adobe Experience Platform Stream connector to sink the Kafka topic (user profile) data into Adobe Experience Platform in real-time. This connector library streams JSON events from Kafka directly into the Adobe Experience Platform in real-time.
The AEP Sink connector needs to consume data from Kafka topic which stores messages in AVRO data type. Hence, we had created a custom message transform to convert the AVRO to JSON datatype, filter’s messages on the field values and it include the static fields needed in the payload. This SMT has helped application team to avoid need for an intermediate consumer to process the data (convert, filter and update fields)
Use Case 2 – HTTP Sink connector
The Digital experience domain uses two HTTP connectors to sink the Kafka topic (mobile-guest-services-push-notifications) data to send push notifications to the guests & (privacy-notifications) manage customer data request to access and delete personal customer data from Adobe Experience Cloud applications, facilitating automated compliance with legal and organizational privacy regulations.
The HTTP sink connector needs to filter the topic messages based on some field values and transform the payload based on the HTTP Endpoints (API). This has been achieved using the custom message transform created for these connectors. This SMT has helped the application team to avoid the need for intermediate consumer/ Endpoint customizations to handle/process the data (filter and update fields).
Use Case 3 – IBM MQ Source connector
The reservation data is sourced through the IBM MQ channels from various systems and third-party applications which is produced into the kafka topic using the IBM MQ source connector which is then consumed by various downstream applications.
The reservation processed by connector (reconciliation) was required for the audit purpose. We had created a custom SMT to validate the message (JSON format) and logged the details needed (reservation id and date time) to identify the unique reservation data for the audit purpose.
