Last modified: Jun 10, 2024

Kafka

Kafka is used as a message broker across the components that make up the Notification solution. We are currently running Apache Kafka on Confluent Cloud through Azure enabling seamless integration with the rest of our components hosted in Azure.

Kafka topic overview

Below is an overview of the Kafka topics used in our system, along with information about the producers and the content of each topic

Health

Description: A topic dedicated to verifying the connectivity between microservices and the Kafka cluster.

Event trigger: External party has requested that the health of the microservice is checked.

Producer: Altinn Notifications, KafkaHealthCheck

Content:

  • Format: string
  • Description: A string not intended to carry any significant data. Default value: test.

Description: A topic dedicated to verifying the connectivity between microservices and the Kafka cluster.

Event trigger: External party has requested that the health of the microservice is checked.

Producer: Altinn Notifications Email, KafkaHealthCheck

Content:

  • Format: string
  • Description: A string not intended to carry any significant data. Default value: test.

Orders

Description: A topic dedicated to orders where the requested send time is about to pass, or has passed.

Event trigger: Requested send time of orders is about to, or has, passed.

Producer: Altinn Notifications, OrderProcessingService

Content:

  • Format: json
  • Data structure: NotificationOrder
  • Description: An order containing notification templates along with complete or partial recipient data.

Description: A topic dedicated to orders where the requested send time is about to pass,or has passed and processing has failed at least once.

Event trigger: Initial processing of the order after passed send time failed due to an unknown or intermittent reason.

Producer: Altinn Notifications, PastDueOrdersConsumer

Content:

  • Format: json
  • Data structure: NotificationOrder
  • Description: An order containing notification templates along with complete or partial recipient data.

Emails

Description: A topic dedicated to emails that are completed and ready to be sent to out.

Event trigger: All required information has been retrieved and populated to the email

Producer: Altinn Notifications, EmailNotificationService

Content:

  • Format: json
  • Data structure: Email
  • Description: An email with all required properties present

Description: A topic dedicated to emails that are completed and ready to be sent to out where at least one previous attempt of sending the email has failed.

Event trigger: Initial attempt to send the email has failed due to an unknown or intermittent reason.

Producer: Altinn Notifications Email, SendEmailQueueConsumer

Content:

  • Format: json
  • Data structure: Email
  • Description: An email with all required properties present.

Description: A topic dedicated to send operation identifiers for emails that have been accepted by Azure Communication services.

Event trigger: The system would like an update on the status for an email that was accepted by Azure Communication Services.

Producer: Altinn Notifications Email, SendingService and StatusService

Content:

  • Format: json
  • Data structure: SendNotificationOperationIdentifier
  • Description: An object grouping notification id, Azure Communication Services operation Id, and the date and time for the last status check.

Description: A topic dedicated to hold updates on the send status of an email notification

Event trigger: An update on the progress of sending an email notification has been received from Azure Communication Services.

Producer: Altinn Notifications Email, StatusService

Content:

Sms

Description: A topic dedicated to sms that are completed and ready to be sent to out.

Event trigger: All required information has been retrieved and populated to the sms

Producer: Altinn Notifications, SmsNotificationService

Content:

  • Format: json
  • Data structure: Sms
  • Description: An sms with all required properties present

Description: A topic dedicated to sms messages that are completed and ready to be sent to out where at least one previous attempt of sending the sms has failed.

Event trigger: Initial attempt to send the sms has failed due to an unknown or intermittent reason.

Producer: Altinn Notifications Sms, SendSmsQueueConsumer

Content:

  • Format: json
  • Data structure: Sms
  • Description: An sms with all required properties present.

Description: A topic dedicated to hold updates on the send status of an sms notification

Event trigger: An update on the progress of sending an sms notification has been received from Link Mobility.

Producer: Altinn Notifications Sms, StatusService

Content:

  • Format: json
  • Data structure: SendOperationResult
  • Description: An object containing the SmsSendResult for a given notification and gatewatyreference to Link Mobility for the transaction.

Platform services

Description: A topic dedicated to hold updates on the Altinn platform components

Event trigger: An change in the state or availability of a platform service has occurred

Producer:

  • Altinn Notifications Email, SendingService

Content:

  • Format: json
  • Data structure: GenericServiceUpdate
  • Description: An object containing an Altinn service update of a schema specified in the payload.

Cluster configuration

The kafka cluster used by Altinn Notifications, is a cluster shared between multiple Altinn components. Configuration in relation to roles and topic policies are in place to ensure integrity in the data on topics owned by the Notifications solution.

Roles

Three roles are defined and used by entities interacting with Kafka from the solution.

  • NotificationsConsumer: can consume from topics following the naming patterns altinn.notifications.* and altinn.platform.*
  • NotificationsProducer : can write to topics following the naming patterns altinn.notifications.* and altinn.platform.*
  • ClusterAdmin: can create a new topics following the naming patterns altinn.notifications.* and altinn.platform.*

Topic configuration

Partitions

The number of partitions on a topic is set at creation. As our solution has the ability to auto scale, we have set the number of partitions equal to the maximum number of nodes available in the AKS cluster running the consumer microservices. This means the maximum numbers of consumers per consumer group will be equal to the number of nodes in the cluster.

Retention

Retention time varies between the environments. For all AT environments retention time for a message is 7 days. In YT, TT and production retention time for a message is 365 days.