Mapping Consumer Offsets between Clusters with Different Message Order
Hey All, looking for some advice on how (if at all) to accomplish this use case.
Scenario: I have two topics of the same name in different clusters. Some replication is happening such that each topic will contain the same messages, but the ordering within them might be different (replication lag). My goal is to sync consumer group offsets such that an active consumer in one would be able to fail over and resume from the other cluster. However, since the message ordering is different, I can't just take the offset from the original cluster and map it directly (since a message that hasn't been consumed yet in cluster 1 could have a smaller offset in cluster 2 than the current offset in cluster 1).
It seems like Kafka Streams might help here, but I haven't used it before and looking to get a sense as to whether this might be viable. In theory, I could have to streams/tables that represent the topic in each cluster, and I'm wondering if there's a way I can dynamically query/window them based on the consumer offset in cluster 1 to identify any messages in cluster 2 that haven't yet appeared in cluster 1 as of the current consumer offset. If such messages exist, the lowest offset would become the consumers offset in cluster 2, and if they don't, I could just use cluster 1's offset.
Any thoughts or suggestions would be greatly appreciated.