Is it possible to have retry mechanism for listeners of 1 listenercontainerfactory and not of another listenercontainerfactory in same application?

[Malhar-Limaye-Zero]

Hi,
I have an existing listener container factory with some configurations and a retry topic configuration bean. The listeners of this container factory are used with named topics like topic1,topic2 etc, so I can include and exclude topics in the configuration as needed.

But the new listener factory(slightly different configurations) has a listener that listens to all topics of topic pattern .*-metrics.

My application fails to start with a No Topics Provided for RetryTopicConfiguration.
Is there a way to not let the retry configuration apply to the second listener factory at all?