OTEL Collector 无法连接到 Elastic APM |码头工人

分享于2022年10月08日 docker elastic-apm elastic-stack elasticsearch open-telemetry 问答
【问题标题】:OTEL Collector not able to connect to Elastic APM | DockerOTEL Collector 无法连接到 Elastic APM |码头工人
【发布时间】:2022-08-01 07:00:35
【问题描述】:

描述错误 OTEL 收集器无法连接到 elastic-apm

重现步骤

请按照此处的说明进行操作,这是一个简单的应用程序 https://github.com/sathishsoundharajan/otel-elastic-apm

你期待看到什么? 我希望看到跟踪日志同时发送到 zipkin 和 elastic-apm 服务器。

你看到了什么?

otel-collector    | 2022-07-29T14:39:44.207Z    error   exporterhelper/queued_retry.go:149  Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors  {"kind": "exporter", "data_type": "traces", "name": "elastic", "error": "sending event request failed: Post \"http://elastic-apm:8200/intake/v2/events\": dial tcp 192.168.32.6:8200: connect: connection refused", "errorVerbose": "Post \"http://elastic-apm:8200/intake/v2/events\": dial tcp 192.168.32.6:8200: connect: connection refused\nsending event request failed\ngo.elastic.co/apm/transport.(*HTTPTransport).sendStreamRequest\n\tgo.elastic.co/apm@v1.15.0/transport/http.go:292\ngo.elastic.co/apm/transport.(*HTTPTransport).SendStream\n\tgo.elastic.co/apm@v1.15.0/transport/http.go:282\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.(*elasticExporter).sendEvents\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter@v0.56.0/exporter.go:197\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.(*elasticExporter).ExportResourceSpans\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter@v0.56.0/exporter.go:149\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.newElasticTracesExporter.func1\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter@v0.56.0/exporter.go:53\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).export\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:70\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/common.go:225\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:147\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:83\ngo.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:113\ngo.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces\n\tgo.opentelemetry.io/collector@v0.56.0/consumer/traces.go:36\ngo.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces\n\tgo.opentelemetry.io/collector@v0.56.0/service/internal/fanoutconsumer/traces.go:75\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export\n\tgo.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:262\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems\n\tgo.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:176\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle\n\tgo.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:143\nruntime.goexit\n\truntime/asm_amd64.s:1571"}
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:149
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:83
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:113
otel-collector    | go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/consumer/traces.go:36
otel-collector    | go.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/service/internal/fanoutconsumer/traces.go:75
otel-collector    | go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:262
otel-collector    | go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:176
otel-collector    | go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:143

码头工人撰写

version: '3'
services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.23
    container_name: elasticsearch
    environment:
      - xpack.security.enabled=false
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -vq '\"status\":\"red\"'"]
      retries: 10
      interval: 20s
    networks:
      - opentelemetry

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:6.8.23
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - 5601:5601
    depends_on:
      elasticsearch:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "--write-out", "'HTTP %{http_code}'", "--silent", "--output", "/dev/null", "http://localhost:5601/"]
      retries: 10
      interval: 10s
    networks:
      - opentelemetry

  elastic-apm:
    container_name: elastic-apm
    build:
      context: ../
      dockerfile: ./build/elastic-apm/Dockerfile
    ports:
      - 8200:8200
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - DAC_OVERRIDE
      - SETGID
      - SETUID
    depends_on:
      elasticsearch:
        condition: service_healthy
      kibana:
        condition: service_healthy
    healthcheck:
      interval: 10s
      retries: 12
      test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/
    networks:
      - opentelemetry

  zipkin:
    image: openzipkin/zipkin
    container_name: zipkin
    ports:
      - 9411:9411
    networks:
      - opentelemetry

  otel-collector:
    container_name: otel-collector
    build:
      context: ../
      dockerfile: ./build/collector/Dockerfile
    ports:
      - 4318:4318
      - 13133:13133
    networks:
      - opentelemetry

volumes:
  elasticsearch-data:
    driver: local

networks:
  opentelemetry:
    driver: bridge 

apm-server.yml

apm-server:
  rum:
    enabled: true
  kibana:
    enabled: true
    host: "kibana:5601"

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

setup.kibana.host: "kibana:5601"
setup.kibana.ssl.enabled: false

环境 码头工人

附加上下文 我想要做的是设置小型 node.js 应用程序

  1. 通过 OTEL 收集器收集迹线
  2. 确保 OTEL 收集器将数据发送到 Zipkin 以及 Elastic APM

    从主机卷曲

    otel-elastic-apm/build on  main [!] on ???? v20.10.17 took 3s
    ➜ curl http://localhost:8200/
    curl: (52) Empty reply from server
    

    在 apm-server 内卷曲

    otel-elastic-apm/build on  main [!] on ???? v20.10.17
    ➜ docker ps
    CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                         NAMES
    576b44ad7b96   build_elastic-apm                                      "/usr/local/bin/dock…"   3 minutes ago   Up 3 minutes (healthy)   0.0.0.0:8200->8200/tcp                                                        elastic-apm
    25e1dd38fb85   docker.elastic.co/kibana/kibana:6.8.23                 "/usr/local/bin/kiba…"   4 minutes ago   Up 4 minutes (healthy)   0.0.0.0:5601->5601/tcp                                                        kibana
    a7b72d802380   docker.elastic.co/elasticsearch/elasticsearch:6.8.23   "/usr/local/bin/dock…"   4 minutes ago   Up 4 minutes (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                elasticsearch
    76f5e0700e10   openzipkin/zipkin                                      "start-zipkin"           4 minutes ago   Up 4 minutes (healthy)   9410/tcp, 0.0.0.0:9411->9411/tcp                                              zipkin
    09c318ae9118   build_otel-collector                                   "/otelcol-contrib --…"   4 minutes ago   Up 4 minutes             0.0.0.0:4318->4318/tcp, 4317/tcp, 55678-55679/tcp, 0.0.0.0:13133->13133/tcp   otel-collector
    
    otel-elastic-apm/build on  main [!] on ???? v20.10.17
    ➜ docker exec -it 576b44ad7b96 sh
    sh-4.2$ curl http://localhost:8200/
    {"ok":{"build_date":"2020-07-09T18:01:36Z","build_sha":"dda44e48cd9c3bac7e5a359a11974594d1d10347","version":"6.8.11"}}sh-4.2$
    

    另一个奇怪的事情:

    我可以看到运行状况检查的 docker-compose 日志正在运行。

    otel-elastic-apm/build on  main [!] on ???? v20.10.17
    ➜ docker ps
    CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                         NAMES
    576b44ad7b96   build_elastic-apm                                      "/usr/local/bin/dock…"   5 minutes ago   Up 5 minutes (healthy)   0.0.0.0:8200->8200/tcp                                                        elastic-apm
    25e1dd38fb85   docker.elastic.co/kibana/kibana:6.8.23                 "/usr/local/bin/kiba…"   6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:5601->5601/tcp                                                        kibana
    a7b72d802380   docker.elastic.co/elasticsearch/elasticsearch:6.8.23   "/usr/local/bin/dock…"   6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                elasticsearch
    76f5e0700e10   openzipkin/zipkin                                      "start-zipkin"           6 minutes ago   Up 6 minutes (healthy)   9410/tcp, 0.0.0.0:9411->9411/tcp                                              zipkin
    09c318ae9118   build_otel-collector                                   "/otelcol-contrib --…"   6 minutes ago   Up 6 minutes             0.0.0.0:4318->4318/tcp, 4317/tcp, 55678-55679/tcp, 0.0.0.0:13133->13133/tcp   otel-collector
    
    otel-elastic-apm/build on  main [!] on ???? v20.10.17
    ➜ docker logs 576b44ad7b96
    2022-07-30T07:40:18.218Z    INFO    instance/beat.go:611    Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]
    2022-07-30T07:40:18.221Z    INFO    instance/beat.go:618    Beat UUID: 95087cff-e948-4969-90fc-ddc9d925f877
    2022-07-30T07:40:18.222Z    INFO    [seccomp]   seccomp/seccomp.go:116  Syscall filter successfully installed
    2022-07-30T07:40:18.222Z    INFO    [beat]  instance/beat.go:931    Beat info   {"system_info": {"beat": {"path": {"config": "/usr/share/apm-server", "data": "/usr/share/apm-server/data", "home": "/usr/share/apm-server", "logs": "/usr/share/apm-server/logs"}, "type": "apm-server", "uuid": "95087cff-e948-4969-90fc-ddc9d925f877"}}}
    2022-07-30T07:40:18.223Z    INFO    [beat]  instance/beat.go:940    Build info  {"system_info": {"build": {"commit": "dda44e48cd9c3bac7e5a359a11974594d1d10347", "libbeat": "6.8.11", "time": "2020-07-09T18:01:36.000Z", "version": "6.8.11"}}}
    2022-07-30T07:40:18.223Z    INFO    [beat]  instance/beat.go:943    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}}
    2022-07-30T07:40:18.224Z    INFO    [beat]  instance/beat.go:947    Host info   {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-07-29T19:44:59Z","containerized":false,"name":"576b44ad7b96","ip":["127.0.0.1/8","192.168.112.6/20"],"kernel_version":"5.10.104-linuxkit","mac":["02:42:c0:a8:70:06"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
    2022-07-30T07:40:18.224Z    INFO    [beat]  instance/beat.go:976    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","setgid","setuid"],"ambient":null}, "cwd": "/usr/share/apm-server", "exe": "/usr/share/apm-server/apm-server", "name": "apm-server", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-07-30T07:40:17.130Z"}}}
    2022-07-30T07:40:18.224Z    INFO    instance/beat.go:280    Setup Beat: apm-server; Version: 6.8.11
    2022-07-30T07:40:18.225Z    INFO    elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch:9200
    2022-07-30T07:40:18.225Z    INFO    [publisher] pipeline/module.go:110  Beat name: 576b44ad7b96
    2022-07-30T07:40:18.225Z    INFO    [beater]    beater/beater.go:91 Falling back to elasticsearch output for sourcemap storage
    2022-07-30T07:40:18.226Z    INFO    [beater]    beater/beater.go:112    No pipeline callback registered
    2022-07-30T07:40:18.226Z    INFO    instance/beat.go:402    apm-server start running.
    2022-07-30T07:40:18.226Z    INFO    [beater]    beater/beater.go:241    self instrumentation is disabled
    2022-07-30T07:40:18.228Z    INFO    [beater]    beater/beater.go:148    host resolved from localhost:8200 to 127.0.0.1:8200
    2022-07-30T07:40:18.228Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/metrics added to request handler
    2022-07-30T07:40:18.228Z    INFO    [handler]   beater/common_handlers.go:163   Path /assets/v1/sourcemaps added to request handler
    2022-07-30T07:40:18.229Z    INFO    elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch:9200
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/rum/sourcemaps added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/rum/errors added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/client-side/sourcemaps added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/transactions added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/client-side/transactions added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/rum/transactions added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/errors added to request handler
    2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/client-side/errors added to request handler
    2022-07-30T07:40:18.230Z    INFO    [handler]   beater/common_handlers.go:169   Path /intake/v2/events added to request handler
    2022-07-30T07:40:18.230Z    INFO    [handler]   beater/common_handlers.go:169   Path /intake/v2/rum/events added to request handler
    2022-07-30T07:40:18.230Z    INFO    [server]    beater/server.go:76 Starting apm-server [dda44e48cd9c3bac7e5a359a11974594d1d10347 built 2020-07-09 18:01:36 +0000 UTC]. Hit CTRL-C to stop it.
    2022-07-30T07:40:18.231Z    INFO    [server]    beater/server.go:77 Listening on: 127.0.0.1:8200
    2022-07-30T07:40:18.231Z    INFO    [server]    beater/server.go:80 RUM endpoints enabled!
    2022-07-30T07:40:18.231Z    INFO    [server]    beater/server.go:97 SSL disabled.
    2022-07-30T07:40:18.231Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "9da73838-38e5-473b-b89a-807a151b8d09", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "", "response_code": 200}
    2022-07-30T07:40:18.232Z    INFO    [onboarding]    beater/onboarding.go:29 Publishing onboarding document
    2022-07-30T07:40:19.233Z    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticsearch:9200))
    2022-07-30T07:40:19.305Z    INFO    elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.8.23
    2022-07-30T07:40:19.325Z    INFO    template/load.go:128    Template already exists and will not be overwritten.
    2022-07-30T07:40:19.325Z    INFO    instance/beat.go:889    Template successfully loaded.
    2022-07-30T07:40:19.325Z    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
    2022-07-30T07:40:28.266Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "82fe1be3-5aaf-42d7-9315-02a677a6e8e1", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:40:38.378Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "3657ba8c-624f-4a54-bbea-c9448d6960a0", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:40:48.448Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "0b8d1559-524e-41ef-9ee2-afb869b4ef7b", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:40:58.550Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "88ec8af0-fa90-4977-9157-e72a472034b1", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:41:08.626Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "fd44355c-372a-41b8-a36b-bdaadbb36baa", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:41:18.725Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "726154c7-8b95-4170-8de1-d6901d534346", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:41:28.821Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "62373e63-f71f-4cc0-9c30-05e185526b98", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:41:38.907Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "2e196be1-46a1-4805-8b90-d2884196b530", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:41:49.012Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "e8c2c571-4ea8-477e-9581-5b7d1014c526", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:41:59.127Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "c6bc6637-6d4e-4d13-85e7-e21ae965dfdc", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:42:09.195Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "76bc029f-2831-410d-9069-a49f78b0de17", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:42:19.300Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "eee825b5-3545-496b-b15a-aeb876ca6f19", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:42:29.406Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "ada4dec2-45df-4517-9417-24d9ac96bb23", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:42:39.486Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "4be275b8-c8fa-4129-b92d-69f460b1a5ea", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:42:49.601Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "59756a31-d187-44ec-ac54-d9041c18b1da", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:42:59.715Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "734cdf3d-4e10-4810-9b69-41936de27788", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:43:09.807Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "69afa8e6-4f85-40c6-aab1-c9515d4bcf67", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:43:19.924Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "66110393-e208-4de3-97f1-470049a87a93", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:43:30.046Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "49bd7707-6071-44ff-92fe-0471b46f861c", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:43:40.140Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "96f06d42-c2ef-420f-b26c-80bf604d4baa", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:43:50.242Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "6a1e5703-bbc9-4280-b278-78e0b5a9922d", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:00.339Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "08709ff6-bb09-49f5-aee1-469443bab827", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:08.800Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "96201795-cc5d-4726-b1d6-87594036f7b5", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:10.414Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "40848199-1d34-4c1f-9292-207a2158dba1", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:20.524Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "cbd6b50c-20b0-460a-9653-327f7c975604", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:30.618Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "36d94913-76c6-4800-9429-a46dd8758a24", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:40.694Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "8cd3b68d-73ce-4d8f-a126-3b01145a935d", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:44:50.789Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "4ae17fad-65de-4719-b3c6-e707239b2ca0", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:45:00.893Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "9c181bba-509b-4fd1-81c8-5e474681c8e3", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:45:10.989Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "b70adb0b-7bd7-4f2f-871e-612e276c5256", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:45:21.091Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "639b21fd-9bb4-4d7e-b51a-13c93c30a5c6", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:45:31.187Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "ee7ceff9-8863-4e70-8a99-98135a81e99a", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:45:41.264Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "b1dcd689-3c03-4e83-b0ed-9ade4bbda7d9", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:45:51.361Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "c82e0fc0-f9d4-4097-a158-317d3427784a", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:46:01.478Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "6ad63614-a1d0-416d-9a9d-636a68210c00", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    2022-07-30T07:46:11.549Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "f65fcc6f-8100-46e8-bdee-1f859722e5e8", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
    

  • 1、OTel采集器的配置是什么?因为 IMO 故障是从收集器到 APM 服务器。
  • 2. 使用这么古老的 Elastic Stack 版本有什么原因吗? 6.x 不再受支持。此外,更多当前版本的 APM 服务器本身支持 OTel 协议(这应该会简化配置并使 OTel 收集器可选)
  • @xeraa 1. 让我们离开 OTEL Collector。我的问题是在 docker 中运行的 APM 服务器无法在 docker 外部访问(主机,试图从主机本身卷曲不起作用)
  • @xeraa 2. 我们的生产系统在 6.x 中运行,我不能简单地升级系统。我也不想单独使用 Elastic APM,就像我上面提到的那样,我想同时使用 Zipkin 和 Elastic APM。所以 OTEL Collector 是最好的方法

【解决方案1】:

使用自定义 Dockerfile 和配置,您是否覆盖了所有接口的默认绑定?来自6.8: https://github.com/elastic/apm-server/blob/6.8/apm-server.docker.yml#L7

这看起来很可疑: Listening on: 127.0.0.1:8200

PS:你还是要升级到原生支持OTel协议的更高版本的APM服务器。您可以继续使用 OTel 收集器,但您不必依赖自定义导出器(我们是 IMO 也不再维护)。

【讨论】: