Kafka
Kafka is used as a message broker across the components that make up the Notification solution. We are currently running Apache Kafka on Confluent Cloud through Azure enabling seamless integration with the rest of our components hosted in Azure.
Kafka topic overview
Below is an overview of the Kafka topics used in our system,
along with information about the producers and the content of each topic
Health
Description: A topic dedicated to verifying the connectivity between microservices and the Kafka cluster.
Event trigger: External party has requested that the health of the microservice is checked.
Producer: Altinn Notifications, KafkaHealthCheck
Content:
- Format: string
- Description: A string not intended to carry any significant data. Default value: test.
Description: A topic dedicated to verifying the connectivity between microservices and the Kafka cluster.
Event trigger: External party has requested that the health of the microservice is checked.
Producer: Altinn Notifications Email, KafkaHealthCheck
Content:
- Format: string
- Description: A string not intended to carry any significant data. Default value: test.
Orders
Description: A topic dedicated to orders where the requested send time is about to pass, or has passed.
Event trigger: Requested send time of orders is about to, or has, passed.
Producer: Altinn Notifications, OrderProcessingService
Content:
- Format: json
- Data structure: NotificationOrder
- Description: An order containing notification templates along with complete or partial recipient data.
Description: A topic dedicated to orders where the requested send time is about to pass,or has passed
and processing has failed at least once.
Event trigger: Initial processing of the order after passed send time failed due to an unknown or intermittent reason.
Producer: Altinn Notifications, PastDueOrdersConsumer
Content:
- Format: json
- Data structure: NotificationOrder
- Description: An order containing notification templates along with complete or partial recipient data.
Emails
Description: A topic dedicated to emails that are completed and ready to be sent to out.
Event trigger: All required information has been retrieved and populated to the email
Producer: Altinn Notifications, EmailNotificationService
Content:
- Format: json
- Data structure: Email
- Description: An email with all required properties present
Description: A topic dedicated to emails that are completed and ready to be sent to out where at least
one previous attempt of sending the email has failed.
Event trigger: Initial attempt to send the email has failed due to an unknown or intermittent reason.
Producer: Altinn Notifications Email, SendEmailQueueConsumer
Content:
- Format: json
- Data structure: Email
- Description: An email with all required properties present.
Description: A topic dedicated to send operation identifiers for emails that have been accepted by Azure Communication services.
Event trigger: The system would like an update on the status for an email that was accepted by Azure Communication Services.
Producer: Altinn Notifications Email, SendingService and StatusService
Content:
- Format: json
- Data structure: SendNotificationOperationIdentifier
- Description: An object grouping notification id, Azure Communication Services operation Id, and the date and time
for the last status check.
Description: A topic dedicated to hold updates on the send status of an email notification
Event trigger: An update on the progress of sending an email notification has been received from Azure Communication Services.
Producer: Altinn Notifications Email, StatusService
Content:
Sms
Description: A topic dedicated to sms that are completed and ready to be sent to out.
Event trigger: All required information has been retrieved and populated to the sms
Producer: Altinn Notifications, SmsNotificationService
Content:
- Format: json
- Data structure: Sms
- Description: An sms with all required properties present
Description: A topic dedicated to sms messages that are completed and ready to be sent to out where at least
one previous attempt of sending the sms has failed.
Event trigger: Initial attempt to send the sms has failed due to an unknown or intermittent reason.
Producer: Altinn Notifications Sms, SendSmsQueueConsumer
Content:
- Format: json
- Data structure: Sms
- Description: An sms with all required properties present.
Description: A topic dedicated to hold updates on the send status of an sms notification
Event trigger: An update on the progress of sending an sms notification has been received from Link Mobility.
Producer: Altinn Notifications Sms, StatusService
Content:
- Format: json
- Data structure: SendOperationResult
- Description: An object containing the SmsSendResult
for a given notification and gatewatyreference to Link Mobility for the transaction.
Cluster configuration
The kafka cluster used by Altinn Notifications, is a cluster shared between multiple Altinn components.
Configuration in relation to roles and topic policies are in place to ensure integrity in the data on topics
owned by the Notifications solution.
Roles
Three roles are defined and used by entities interacting with Kafka from the solution.
- NotificationsConsumer: can consume from topics following the naming patterns altinn.notifications.* and altinn.platform.*
- NotificationsProducer : can write to topics following the naming patterns altinn.notifications.* and altinn.platform.*
- ClusterAdmin: can create a new topics following the naming patterns altinn.notifications.* and altinn.platform.*
Topic configuration
Partitions
The number of partitions on a topic is set at creation. As our solution has the ability to auto scale,
we have set the number of partitions equal to the maximum number of nodes available in the AKS cluster
running the consumer microservices. This means the maximum numbers of consumers per consumer group will be equal to
the number of nodes in the cluster.
Retention
Retention time varies between the environments.
For all AT environments retention time for a message is 7 days.
In YT, TT and production retention time for a message is 365 days.
