Bad KAFKA_ADVERTISED_LISTENERS when exposing host ports to the container #3187

Activity

1. stale commented on Dec 19, 2020

   

   staleBot

   on Dec 19, 2020

   More actions

   This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.
2. stale

   added

   stale

   on Dec 19, 2020
3. NersesAM commented on Apr 19, 2021

   

   NersesAM

   on Apr 19, 2021

   More actions

   I've got the same problem.
   Kafka fails to start with the same exception posted by @goncalonsp

   I was able to create a small test that reproduces the issue.

   import org.junit.jupiter.api.Test;
   import org.testcontainers.Testcontainers;
   import org.testcontainers.containers.KafkaContainer;
   import org.testcontainers.containers.Network;
   
   public class KafkaContainerBug {
   
       @Test
       public void reproduceBug() {
           Testcontainers.exposeHostPorts(3454);
           try (
                   Network bridgeNetwork = Network.builder().driver("bridge").build();
                   KafkaContainer kafka = new KafkaContainer().withNetwork(bridgeNetwork).withNetworkAliases("kafka")
           ) {
               kafka.start();
           }
       }
   }
   

   without Testcontainers.exposeHostPorts(3454); it works. When this line is added 2nd network keyed "bridge" is added to containerInfo.getNetworkSettings().getNetworks() map.

   But in general https://github.com/testcontainers/testcontainers-java/blob/master/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L128-L129

   containerInfo.getNetworkSettings().getNetworks().values().stream()
                       .map(it -> "BROKER://" + it.getIpAddress() + ":9092")
   

   will never work for more than one networks.

   Any suggestions how to fix it?
4. stale

   removed

   stale

   on Apr 19, 2021
5. worldtiki commented on Apr 20, 2021

   

   worldtiki

   on Apr 20, 2021

   Contributor

   More actions

   I'm trying to understand the need for the bridge network.

   If you want to access the Kafka container from a non containerized app you can:

   KafkaContainer kafkaContainer = new KafkaContainer();
   

   And in your app:

   bootstrapServers = kafkaContainer.getBootstrapServers()
   

   If you want to access from other containers you can:

   Network network = Network.newNetwork();
   
   KafkaContainer kafkaContainer = new KafkaContainer()
                   .withNetworkAliases("broker")
                   .withNetwork(network);
   kafkaContainer.start();
   

   And then for the configs of the other containerized app, add it to the same network and use the alias like this:

   bootstrapServers = "PLAINTEXT://broker:9092";
   

   👍React with 👍1Reacted by Victoria Shabunia

   👎React with 👎2Reacted by jameswylyreasint and Vilkesh Patel
6. NersesAM commented on Apr 23, 2021

   

   NersesAM

   on Apr 23, 2021

   More actions

   @worldtiki my use-case is I need to connect from inside docker container to my test app running on the host.
7. botismarius commented on Apr 26, 2021

   

   botismarius

   on Apr 26, 2021

   Last edited by botismarius

   More actions

   Hi,

   I am also unable to use the KafkaContainer with a custom network while exposing some other port. My use-case is:

   1. I need to expose the local port 3306 (MySQL) to toxiproxy testcontainer;
   2. I use the same network both for toxiproxy container and for kafka container.

   The error I get is:

   ERROR Fatal error during SupportedServerStartable startup. Prepare to shutdown (io.confluent.support.metrics.SupportedKafka)
   java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://localhost:53510,BROKER://192.168.128.3:9092,BROKER://172.17.0.6:9092
   

   If I omit this line from my setup, it works: Testcontainers.exposeHostPorts(3306).

   Please advise.
8. botismarius commented on Apr 26, 2021

   

   botismarius

   on Apr 26, 2021

   Last edited by botismarius

   More actions

   As a workaround I am extending the KafkaContainer class and removing the bridge network:

   class KafkaContainerWithoutBridge extends KafkaContainer {
   
       public KafkaContainerWithoutBridge(DockerImageName dockerImageName) {
           super(dockerImageName);
       }
   
       @Override
       protected void containerIsStarting(InspectContainerResponse containerInfo, boolean reused) {
           containerInfo.getNetworkSettings().getNetworks().remove("bridge");
           super.containerIsStarting(containerInfo, reused);
       }
   }
   
9. NersesAM commented on Apr 29, 2021

   

   NersesAM

   on Apr 29, 2021

   Last edited by NersesAM

   More actions

   There is a Kafka ticket that will make this bug go away when it is resolved. But I assume it doesn't have high enough priority, so unlikely it will be picked up soon.

   https://issues.apache.org/jira/browse/KAFKA-9257

   👍React with 👍2Reacted by Sergei Egorov and Marius Botis
10. bsideup commented on Apr 29, 2021

    

    bsideup

    on Apr 29, 2021

    Member

    More actions

    @NersesAM thanks for sharing this link! We will try to reach out to the Kafka folks and also workaround it on our side 👍
11. NersesAM commented on Apr 29, 2021

    

    NersesAM

    on Apr 29, 2021

    More actions

    @NersesAM thanks for sharing this link! We will try to reach out to the Kafka folks and also workaround it on our side 👍

    Thanks Sergei,

    I think workaround in testcontainers will be needed for some time anyway, because even if Kafka fixes that issue, not everyone will be able to use latest version from the get go.
12. eddumelendez

    added

    modules/kafka

    on Jul 24, 2023
13. aymenjamaaoui commented on Oct 4, 2026

    

    aymenjamaaoui

    on Oct 4, 2026

    More actions

    This looks fixed since 1.16.0 by #4213. The old org.testcontainers.containers.KafkaContainer now uses only the network set with withNetwork (or bridge) for the BROKER listener. Tests testWithHostExposedPort and testWithHostExposedPortAndExternalNetwork in KafkaContainerTest cover this case.

    The new org.testcontainers.kafka.KafkaContainer and ConfluentKafkaContainer use the container hostname for the BROKER listener, so they do not have this problem. I ran the test from this issue (exposeHostPorts + custom bridge network, so the container has 2 networks) on main with both classes, and Kafka starts and works.

    I think this issue can be closed.