D 2024-02-01 23:56:53.093350+00:00 Params: [('source_key', 'agdicmlk...'), ('last_polled', '2024-01-31-22-47-39')]
I 2024-02-01 23:56:53.202552+00:00 Source: imdario (GitHub) imdario, https://brid.gy/github/imdario
I 2024-02-01 23:56:53.202873+00:00 Last poll: https://brid.gy/log?start_time=1706741259&key=agdicmlkLWd5chMLEgZHaXRIdWIiB2ltZGFyaW8M
I 2024-02-01 23:56:53.316605+00:00 Updating imdario (GitHub) /github/imdario : {'poll_status': 'polling', 'last_poll_attempt': datetime.datetime(2024, 2, 1, 23, 56, 53, 202973, tzinfo=datetime.timezone.utc), 'rate_limited': False}
D 2024-02-01 23:56:53.492174+00:00 Using ETag Wed, 31 Jan 2024 22:00:39 GMT, last activity id zeit:hyper:2175
I 2024-02-01 23:56:53.594326+00:00 no refresh_token
I 2024-02-01 23:56:53.594812+00:00 requests.get https://api.github.com/notifications?all=true&participating=true&per_page=10 {'headers': {'If-Modified-Since': '...', 'Authorization': '...'}}
I 2024-02-01 23:56:54.741772+00:00 Received 200
I 2024-02-01 23:56:54.742372+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/pulls/21715 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:55.723313+00:00 Received 200
I 2024-02-01 23:56:55.726833+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/pulls/2482 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:56.407598+00:00 Received 200
I 2024-02-01 23:56:56.410313+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/pulls/2537 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:57.087652+00:00 Received 200
I 2024-02-01 23:56:57.109107+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/pulls/2502 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:57.567726+00:00 Received 200
I 2024-02-01 23:56:57.571010+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/pulls/22502 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:58.272438+00:00 Received 200
I 2024-02-01 23:56:58.276273+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/pulls/22491 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:58.707807+00:00 Received 200
I 2024-02-01 23:56:58.710810+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/pulls/2510 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:56:59.886635+00:00 Received 200
I 2024-02-01 23:56:59.893567+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/pulls/2049 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:00.433074+00:00 Received 200
I 2024-02-01 23:57:00.436453+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/pulls/2066 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:00.934123+00:00 Received 200
I 2024-02-01 23:57:00.937460+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/pulls/1981 {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:01.789369+00:00 Received 200
I 2024-02-01 23:57:01.792536+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/issues/21715/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:02.198223+00:00 Received 200
I 2024-02-01 23:57:02.201638+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/issues/21715/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:03.308482+00:00 Received 200
I 2024-02-01 23:57:03.310669+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2482/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:03.601689+00:00 Received 200
I 2024-02-01 23:57:03.603799+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2482/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:04.089588+00:00 Received 200
I 2024-02-01 23:57:04.094065+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2537/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:04.979622+00:00 Received 200
I 2024-02-01 23:57:04.982440+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2537/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:05.335351+00:00 Received 200
I 2024-02-01 23:57:05.338644+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2502/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:05.637292+00:00 Received 200
I 2024-02-01 23:57:05.645243+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2502/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:06.043010+00:00 Received 200
I 2024-02-01 23:57:06.045239+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/issues/22502/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:06.327974+00:00 Received 200
I 2024-02-01 23:57:06.330036+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/issues/22502/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:06.802959+00:00 Received 200
I 2024-02-01 23:57:06.805908+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/issues/22491/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:07.337123+00:00 Received 200
I 2024-02-01 23:57:07.341661+00:00 requests.get https://api.github.com/repos/DataDog/datadog-agent/issues/22491/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:07.664309+00:00 Received 200
I 2024-02-01 23:57:07.666828+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2510/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:08.238603+00:00 Received 200
I 2024-02-01 23:57:08.242068+00:00 requests.get https://api.github.com/repos/DataDog/dd-trace-go/issues/2510/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:08.663344+00:00 Received 200
I 2024-02-01 23:57:08.666784+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/issues/2049/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:08.967394+00:00 Received 200
I 2024-02-01 23:57:08.969404+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/issues/2049/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:09.396369+00:00 Received 200
I 2024-02-01 23:57:09.407681+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/issues/2066/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:10.280649+00:00 Received 200
I 2024-02-01 23:57:10.282818+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/issues/2066/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:10.677471+00:00 Received 200
I 2024-02-01 23:57:10.682570+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/issues/1981/comments?since=2024-01-31T22:00:39Z {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:11.035256+00:00 Received 200
I 2024-02-01 23:57:11.037663+00:00 requests.get https://api.github.com/repos/DataDog/system-tests/issues/1981/reactions {'headers': {'Authorization': '...'}}
I 2024-02-01 23:57:11.387674+00:00 Received 200
I 2024-02-01 23:57:11.396483+00:00 Found 10 public activities: dict_keys(['tag:github.com,2013:DataDog:datadog-agent:21715', 'tag:github.com,2013:DataDog:dd-trace-go:2482', 'tag:github.com,2013:DataDog:dd-trace-go:2537', 'tag:github.com,2013:DataDog:dd-trace-go:2502', 'tag:github.com,2013:DataDog:datadog-agent:22502', 'tag:github.com,2013:DataDog:datadog-agent:22491', 'tag:github.com,2013:DataDog:dd-trace-go:2510', 'tag:github.com,2013:DataDog:system-tests:2049', 'tag:github.com,2013:DataDog:system-tests:2066', 'tag:github.com,2013:DataDog:system-tests:1981'])
I 2024-02-01 23:57:11.396802+00:00 Found 0 private activities: dict_keys([])
D 2024-02-01 23:57:11.404173+00:00 activity[content] ## Benchmarks
Benchmark execution time: 2024-01-30 23:23:06
Comparing candidate commit 32b50fbc6fde51abf022da5aaabce0d737fba854 in PR branch `munir/implement-span-links-poc` with baseline commit 8c4834f24951876c7f50a98fa640a96541cc3c04 in branch `main`.
Found 0 performance improvements and 14 performance regressions! Performance is the same for 25 metrics, 2 unstable metrics.
#### scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24
- ๐ฅ `allocated_mem` [+73 bytes; +83 bytes] or [+2.039%; +2.315%]
#### scenario:BenchmarkOTelApiWithCustomTags/otel_api-24
- ๐ฅ `allocated_mem` [+136 bytes; +149 bytes] or [+2.792%; +3.059%]
- ๐ฅ `allocations` [+2; +2] or [+4.545%; +4.545%]
#### scenario:BenchmarkPartialFlushing/Disabled-24
- ๐ฅ `allocated_mem` [+9.135MB; +9.354MB] or [+3.010%; +3.083%]
- ๐ฅ `execution_time` [+6.775ms; +12.972ms] or [+2.402%; +4.599%]
#### scenario:BenchmarkPartialFlushing/Enabled-24
- ๐ฅ `allocated_mem` [+9.276MB; +12.925MB] or [+2.402%; +3.347%]
#### scenario:BenchmarkSingleSpanRetention/no-rules-24
- ๐ฅ `allocated_mem` [+4.848KB; +4.849KB] or [+3.087%; +3.088%]
- ๐ฅ `execution_time` [+9.842ยตs; +10.681ยตs] or [+3.967%; +4.306%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24
- ๐ฅ `allocated_mem` [+4.684KB; +4.857KB] or [+2.937%; +3.045%]
- ๐ฅ `execution_time` [+9.284ยตs; +10.797ยตs] or [+3.712%; +4.317%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24
- ๐ฅ `allocated_mem` [+4.629KB; +4.754KB] or [+2.902%; +2.981%]
- ๐ฅ `execution_time` [+9.046ยตs; +10.539ยตs] or [+3.615%; +4.212%]
#### scenario:BenchmarkStartSpan-24
- ๐ฅ `allocated_mem` [+47 bytes; +48 bytes] or [+3.058%; +3.107%]
#### scenario:BenchmarkTracerAddSpans-24
- ๐ฅ `allocated_mem` [+48 bytes; +49 bytes] or [+2.177%; +2.228%]
=> ## Benchmarks
Benchmark execution time: 2024-02-01 19:10:30
Comparing candidate commit c75a0953c52224c41f53341b9acd90e58aef09d8 in PR branch `munir/implement-span-links-poc` with baseline commit e707cc94fdd4e663fccdb6ab3fb3258539444bc0 in branch `main`.
Found 0 performance improvements and 12 performance regressions! Performance is the same for 27 metrics, 2 unstable metrics.
#### scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24
- ๐ฅ `allocated_mem` [+76 bytes; +84 bytes] or [+2.101%; +2.326%]
#### scenario:BenchmarkPartialFlushing/Disabled-24
- ๐ฅ `allocated_mem` [+9.143MB; +9.430MB] or [+3.014%; +3.108%]
#### scenario:BenchmarkPartialFlushing/Enabled-24
- ๐ฅ `allocated_mem` [+9.313MB; +13.147MB] or [+2.411%; +3.404%]
#### scenario:BenchmarkSingleSpanRetention/no-rules-24
- ๐ฅ `allocated_mem` [+4.848KB; +4.849KB] or [+3.087%; +3.087%]
- ๐ฅ `execution_time` [+9.682ยตs; +11.064ยตs] or [+3.901%; +4.457%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24
- ๐ฅ `allocated_mem` [+4.732KB; +4.906KB] or [+2.968%; +3.077%]
- ๐ฅ `execution_time` [+9.372ยตs; +10.696ยตs] or [+3.745%; +4.274%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24
- ๐ฅ `allocated_mem` [+4.700KB; +4.895KB] or [+2.948%; +3.071%]
- ๐ฅ `execution_time` [+8.437ยตs; +10.194ยตs] or [+3.364%; +4.064%]
#### scenario:BenchmarkStartSpan-24
- ๐ฅ `allocated_mem` [+48 bytes; +48 bytes] or [+3.076%; +3.128%]
#### scenario:BenchmarkTracerAddSpans-24
- ๐ฅ `allocated_mem` [+49 bytes; +49 bytes] or [+2.212%; +2.212%]
- ๐ฅ `execution_time` [+85.364ns; +131.036ns] or [+2.110%; +3.239%]
D 2024-02-01 23:57:11.404394+00:00 discovering original posts for: https://github.com/DataDog/datadog-agent/pull/21715
I 2024-02-01 23:57:11.405269+00:00 requests.head https://github.com/DataDog/datadog-agent/labels/team/agent-apm {'headers': {}}
I 2024-02-01 23:57:12.707968+00:00 Received 404
W 2024-02-01 23:57:12.708256+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/labels/team/agent-apm: https://github.com/DataDog/datadog-agent/labels/team/agent-apm
I 2024-02-01 23:57:12.708539+00:00 requests.head https://github.com/DataDog/datadog-agent/labels/7.51.0-drop {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:13.800177+00:00 Received 404
W 2024-02-01 23:57:13.800445+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/labels/7.51.0-drop: https://github.com/DataDog/datadog-agent/labels/7.51.0-drop
I 2024-02-01 23:57:13.800672+00:00 requests.head https://github.com/DataDog/datadog-agent/assets/62392438/f18baba6-d698-4515-ae7e-0aa3d79202c1 {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:14.303054+00:00 Redirected to https://github-production-user-asset-6210df.s3.amazonaws.com/62392438/300623701-f18baba6-d698-4515-ae7e-0aa3d79202c1.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240201T235714Z&X-Amz-Expires=300&X-Amz-Signature=...&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970739
I 2024-02-01 23:57:14.303466+00:00 Received 403
W 2024-02-01 23:57:14.303654+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/assets/62392438/f18baba6-d698-4515-ae7e-0aa3d79202c1: https://github-production-user-asset-6210df.s3.amazonaws.com/62392438/300623701-f18baba6-d698-4515-ae7e-0aa3d79202c1.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240201T235714Z&X-Amz-Expires=300&X-Amz-Signature=...&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970739
I 2024-02-01 23:57:14.304035+00:00 requests.head https://github.com/DataDog/datadog-agent/blob/main/docs/dev/contributing.md#reno {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:14.660275+00:00 Received 200
I 2024-02-01 23:57:14.660618+00:00 requests.head https://github.com/DataDog/documentation/issues/new {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:15.089407+00:00 Redirected to https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FDataDog%2Fdocumentation%2Fissues%2Fnew
I 2024-02-01 23:57:15.089824+00:00 Received 200
D 2024-02-01 23:57:15.089990+00:00 Resolved https://github.com/DataDog/documentation/issues/new to https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FDataDog%2Fdocumentation%2Fissues%2Fnew
I 2024-02-01 23:57:15.090303+00:00 requests.head https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:15.630076+00:00 Received 200
I 2024-02-01 23:57:15.630568+00:00 Original post discovery found original posts set(), mentions {'https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FDataDog%2Fdocumentation%2Fissues%2Fnew', 'https://github.com/DataDog/datadog-agent/blob/main/docs/dev/contributing.md#reno', 'https://github-production-user-asset-6210df.s3.amazonaws.com/62392438/300623701-f18baba6-d698-4515-ae7e-0aa3d79202c1.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240201T235714Z&X-Amz-Expires=300&X-Amz-Signature=...&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970739', 'https://github.com/DataDog/datadog-agent/labels/team/agent-apm', 'https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml', 'https://github.com/DataDog/datadog-agent/labels/7.51.0-drop'}
I 2024-02-01 23:57:15.631884+00:00 requests.head https://github.com/DataDog/datadog-agent/pull/21715 {'headers': None}
I 2024-02-01 23:57:17.024457+00:00 Received 200
I 2024-02-01 23:57:17.024835+00:00 starting posse post discovery with syndicated https://github.com/DataDog/datadog-agent/pull/21715
D 2024-02-01 23:57:17.236330+00:00 activity[content] ## Bloop Bleep... Dogbot Here
# Regression Detector Results
Run ID: e5950a92-0f0b-40f7-b23d-270e9799ef64
Baseline: 4260c809cf7e1bfabb0505c08983cb5fc7730768
Comparison: 0c237632985dd754f4178f689bd6cd00e436cf80
Total CPUs: 7
Performance changes are noted in the **perf** column of each table:
* โ
= significantly better comparison variant performance
* โ = significantly worse comparison variant performance
* โ = no significant change in performance
## No significant changes in experiment optimization goals
**Confidence level:** 90.00%
**Effect size tolerance:** |ฮ mean %| โฅ 5.00%
There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
<details>
<summary><h2>
Experiments ignored for regressions
</h2></summary>
Regressions in experiments with settings containing `erratic: true` are ignored.
| perf | experiment | goal | ฮ mean % | ฮ mean % CI |
|------|-------------------|--------------------|----------|----------------|
| โ | idle | memory utilization | +0.45 | [+0.40, +0.49] |
| โ | file_tree | memory utilization | +0.19 | [+0.08, +0.31] |
| โ | file_to_blackhole | % cpu utilization | -0.09 | [-6.63, +6.45] |
</details>
<details>
<summary><h2>
Fine details of change detection per experiment
</h2></summary>
| perf | experiment | goal | ฮ mean % | ฮ mean % CI |
|------|-----------------------------------------|--------------------|----------|----------------|
| โ | tcp_syslog_to_blackhole | ingress throughput | +0.63 | [+0.57, +0.69] |
| โ | idle | memory utilization | +0.45 | [+0.40, +0.49] |
| โ | process_agent_standard_check | memory utilization | +0.28 | [+0.21, +0.34] |
| โ | file_tree | memory utilization | +0.19 | [+0.08, +0.31] |
| โ | process_agent_standard_check_with_stats | memory utilization | +0.11 | [+0.07, +0.16] |
| โ | trace_agent_json | ingress throughput | +0.05 | [+0.03, +0.08] |
| โ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.05, +0.05] |
| โ | uds_dogstatsd_to_api | ingress throughput | -0.00 | [-0.03, +0.03] |
| โ | trace_agent_msgpack | ingress throughput | -0.03 | [-0.05, -0.02] |
| โ | file_to_blackhole | % cpu utilization | -0.09 | [-6.63, +6.45] |
| โ | process_agent_real_time_mode | memory utilization | -0.13 | [-0.18, -0.09] |
| โ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.37 | [-1.82, +1.08] |
| โ | otel_to_otel_logs | ingress throughput | -1.39 | [-2.11, -0.68] |
</details>
<details>
<summary><h2>
Explanation
</h2></summary>
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "ฮ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
1. Its estimated |ฮ mean %| โฅ 5.00%, indicating the change is big enough to merit a closer look.
2. Its 90.00% confidence interval "ฮ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
3. Its configuration does not mark it "erratic".
</details> => ## Bloop Bleep... Dogbot Here
# Regression Detector Results
Run ID: 73edd747-012c-4f08-9102-6e1c4013003b
Baseline: 23bb19e6b75f5f3c70c6c5ca69d513f9b424954b
Comparison: 042304f1b750449d2df8d0c13a61d305c9a5a496
Total CPUs: 7
Performance changes are noted in the **perf** column of each table:
* โ
= significantly better comparison variant performance
* โ = significantly worse comparison variant performance
* โ = no significant change in performance
## No significant changes in experiment optimization goals
**Confidence level:** 90.00%
**Effect size tolerance:** |ฮ mean %| โฅ 5.00%
There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
<details>
<summary><h2>
Experiments ignored for regressions
</h2></summary>
Regressions in experiments with settings containing `erratic: true` are ignored.
| perf | experiment | goal | ฮ mean % | ฮ mean % CI |
|------|-------------------|--------------------|----------|----------------|
| โ | idle | memory utilization | +0.85 | [+0.83, +0.87] |
| โ | file_to_blackhole | % cpu utilization | +0.42 | [-6.15, +7.00] |
| โ | file_tree | memory utilization | +0.21 | [+0.14, +0.28] |
</details>
<details>
<summary><h2>
Fine details of change detection per experiment
</h2></summary>
| perf | experiment | goal | ฮ mean % | ฮ mean % CI |
|------|-----------------------------------------|--------------------|----------|----------------|
| โ | otel_to_otel_logs | ingress throughput | +4.11 | [+3.34, +4.89] |
| โ | idle | memory utilization | +0.85 | [+0.83, +0.87] |
| โ | file_to_blackhole | % cpu utilization | +0.42 | [-6.15, +7.00] |
| โ | process_agent_standard_check | memory utilization | +0.31 | [+0.26, +0.35] |
| โ | process_agent_real_time_mode | memory utilization | +0.22 | [+0.19, +0.26] |
| โ | process_agent_standard_check_with_stats | memory utilization | +0.21 | [+0.17, +0.25] |
| โ | file_tree | memory utilization | +0.21 | [+0.14, +0.28] |
| โ | trace_agent_msgpack | ingress throughput | +0.02 | [-0.00, +0.04] |
| โ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.06, +0.06] |
| โ | uds_dogstatsd_to_api | ingress throughput | -0.00 | [-0.03, +0.03] |
| โ | trace_agent_json | ingress throughput | -0.02 | [-0.05, +0.01] |
| โ | tcp_syslog_to_blackhole | ingress throughput | -1.16 | [-1.22, -1.09] |
| โ | uds_dogstatsd_to_api_cpu | % cpu utilization | -2.83 | [-4.25, -1.42] |
</details>
<details>
<summary><h2>
Explanation
</h2></summary>
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "ฮ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
1. Its estimated |ฮ mean %| โฅ 5.00%, indicating the change is big enough to merit a closer look.
2. Its 90.00% confidence interval "ฮ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
3. Its configuration does not mark it "erratic".
</details>
I 2024-02-01 23:57:17.236874+00:00 Response changed! Re-propagating. Original: Response(key=Key('Response', 'tag:github.com,2013:DataDog:datadog-agent:IC_kwDOAvp-M85xhrpV'), activities_json=['{"id":"tag:github.com,2013:DataDog:datadog-agent:21715","url":"https://github.com/DataDog/datadog-agent/pull/21715","content":"<!--\\n* New contributors are highly encouraged to read our\\n [CONTRIBUTING](/CONTRIBUTING.md) documentation.\\n* The pull request:\\n * Should only fix one issue or add one feature at a time.\\n * Must update the test suite for the relevant functionality.\\n * Should pass all status checks before being reviewed or merged.\\n* Commit titles should be prefixed with general area of pull request\'s change.\\n\\n-->\\n### What does this PR do?\\nAdd Span Links support to the trace agent\\n<!--\\n* A brief description of the change being made with this pull request.\\n* If the description here cannot be expressed in a succinct form, consider\\n opening multiple pull requests instead of a single one.\\n-->\\n\\n### Motivation\\nSpan links are an important feature for distributed traces in messaging systems or where span relationships aren\'t as simple as the current \\"descends from\\" model.\\n<!--\\n* What inspired you to submit this pull request?\\n* Link any related GitHub issues or PRs here.\\n-->\\n\\n### Additional Notes\\n\\n<!--\\n* Anything else we should know when reviewing?\\n* Include benchmarking information here whenever possible.\\n* Include info about alternatives that were considered and why the proposed\\n version was chosen.\\n-->\\n\\n### Possible Drawbacks / Trade-offs\\n\\n<!--\\n* What are the possible side-effects or negative impacts of the code ... Describe how to test/QA your changes\\n\\n<!--\\n* Write here in detail or link to detailed instructions on how this change can\\n be tested/QAd/validated, including any environment setup.\\n-->\\n\\n### Reviewer\'s Checklist\\n<!--\\n* Authors can use this list as a reference to ensure that there are no problems\\n during the review but the signing off is to be done by the reviewer(s).\\n\\nNote: Adding GitHub labels is only possible for contributors with write access.\\n-->\\n\\n- [x] If known, an appropriate milestone has been selected; otherwise the `Triage` milestone is set.\\n- [x] Use the `major_change` label if your change either has a major impact on the code ... is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.\\n- [x] A [release note](https://github.com/DataDog/datadog-agent/blob/main/docs/dev/contributing.md#reno) has been added or the `changelog/no-changelog` label has been applied.\\n- [ ] Changed code ... automated tests for its functionality.\\n- [ ] Adequate QA/testing plan information is provided. Except if the `qa/skip-qa` label, with required either `qa/done` or `qa/no-code-change` labels, are applied.\\n- [x] At least one `team/..` label has been applied, indicating the team(s) that should QA this change.\\n- [ ] If applicable, docs team has been notified or [an issue has been opened on the documentation repo](https://github.com/DataDog/documentation/issues/new)..\\n- [ ] If applicable, the `need-change/operator` and `need-change/helm` labels have been applied.\\n- [ ] If applicable, the `k8s/<min-version>` label, indicating the lowest Kubernetes version compatible with this feature.\\n- [ ] If applicable, the [config template](https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml) has been updated.\\n"}'], created=DatetimeWithNanoseconds(2024, 1, 27, 12, 23, 38, 815104, tzinfo=datetime.timezone.utc), error=[], failed=[], leased_until=None, old_response_jsons=[], original_posts=[], response_json='{"id":"tag:github.com,2013:DataDog:datadog-agent:IC_kwDOAvp-M85xhrpV","url":"https://github.com/DataDog/datadog-agent/pull/21715#issuecomment-1904654933","author":{"id":"tag:github.com,2013:MDM6Qm90MzQ1NzgxMTY=","url":"https://github.com/apps/pr-commenter","displayName":"pr-commenter[bot]","objectType":"person","username":"pr-commenter[bot]","image":{"url":"https://avatars.githubusercontent.com/u/365230?v=4"}},"content":"## Bloop Bleep... Dogbot Here\\n\\n# Regression Detector Results\\n\\nRun ID: e5950a92-0f0b-40f7-b23d-270e9799ef64\\nBaseline: 4260c809cf7e1bfabb0505c08983cb5fc7730768\\nComparison: 0c237632985dd754f4178f689bd6cd00e436cf80\\nTotal CPUs: 7\\n\\nPerformance changes are noted in the **perf** column of each table:\\n\\n* \\u2705 = significantly better comparison variant performance\\n* \\u274c = significantly worse comparison variant performance\\n* \\u2796 = no significant change in performance\\n\\n## No significant changes in experiment optimization goals\\n\\n**Confidence level:** 90.00%\\n**Effect size tolerance:** |\\u0394 mean %| \\u2265 5.00%\\n\\nThere were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.\\n\\n<details>\\n<summary><h2>\\nExperiments ignored for regressions\\n</h2></summary>\\n\\nRegressions in experiments with settings containing `erratic: true` are ignored.\\n\\n| perf | experiment | goal | \\u0394 mean % | \\u0394 mean % CI |\\n|------|-------------------|--------------------|----------|----------------|\\n| \\u2796 | idle | memory utilization | +0.45 | [+0.40, +0.49] |\\n| \\u2796 | file_tree | memory utilization | +0.19 | [+0.08, +0.31] |\\n| \\u2796 | file_to_blackhole | % cpu utilization | -0.09 | [-6.63, +6.45] |\\n</details>\\n\\n<details>\\n<summary><h2>\\nFine details of change detection per experiment\\n</h2></summary>\\n\\n| perf | experiment | goal | \\u0394 mean % | \\u0394 mean % CI |\\n|------|-----------------------------------------|--------------------|----------|----------------|\\n| \\u2796 | tcp_syslog_to_blackhole | ingress throughput | +0.63 | [+0.57, +0.69] |\\n| \\u2796 | idle | memory utilization | +0.45 | [+0.40, +0.49] |\\n| \\u2796 | process_agent_standard_check | memory utilization | +0.28 | [+0.21, +0.34] |\\n| \\u2796 | file_tree | memory utilization | +0.19 | [+0.08, +0.31] |\\n| \\u2796 | process_agent_standard_check_with_stats | memory utilization | +0.11 | [+0.07, +0.16] |\\n| \\u2796 | trace_agent_json | ingress throughput | +0.05 | [+0.03, +0.08] |\\n| \\u2796 | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.05, +0.05] |\\n| \\u2796 | uds_dogstatsd_to_api | ingress throughput | -0.00 | [-0.03, +0.03] |\\n| \\u2796 | trace_agent_msgpack | ingress throughput | -0.03 | [-0.05, -0.02] |\\n| \\u2796 | file_to_blackhole | % cpu utilization | -0.09 | [-6.63, +6.45] |\\n| \\u2796 | process_agent_real_time_mode | memory utilization | -0.13 | [-0.18, -0.09] |\\n| \\u2796 | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.37 | [-1.82, +1.08] |\\n| \\u2796 | otel_to_otel_logs | ingress throughput | -1.39 | [-2.11, -0.68] |\\n</details>\\n\\n<details>\\n<summary><h2>\\nExplanation\\n</h2></summary>\\n\\nA regression test is an A/B test of target performance in a repeatable rig, where \\"performance\\" is measured as \\"comparison variant minus baseline variant\\" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted \\"\\u0394 mean % CI\\".\\n\\nFor each experiment, we decide whether a change in performance is a \\"regression\\" -- a change worth investigating further -- if all of the following criteria are true:\\n\\n1. Its estimated |\\u0394 mean %| \\u2265 5.00%, indicating the change is big enough to merit a closer look.\\n\\n2. Its 90.00% confidence interval \\"\\u0394 mean % CI\\" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.\\n\\n3. Its configuration does not mark it \\"erratic\\".\\n</details>","published":"2024-01-22T19:22:08+00:00","updated":"2024-01-26T16:07:37+00:00","objectType":"comment","inReplyTo":[{"url":"https://github.com/DataDog/datadog-agent/pull/21715"}]}', sent=[], skipped=[], source=Key('GitHub', 'imdario'), status='complete', type='comment', unsent=[], updated=datetime.datetime(2024, 2, 1, 23, 57, 17, 175249, tzinfo=datetime.timezone.utc), urls_to_activity=None)
I 2024-02-01 23:57:17.436575+00:00 Added propagate task projects/brid-gy/locations/us-central1/queues/propagate/tasks/2337244635464599586 with ETA None: {'app_engine_http_request': {'http_method': 'POST', 'relative_uri': '/_ah/queue/propagate', 'body': b'response_key=agdicmlkLWd5cksLEghSZXNwb25zZSI9dGFnOmdpdGh1Yi5jb20sMjAxMzpEYXRhRG9nOmRhdGFkb2ctYWdlbnQ6SUNfa3dET0F2cC1NODV4aHJwVgw', 'headers': {'Content-Type': 'application/x-www-form-urlencoded'}}}
D 2024-02-01 23:57:17.436890+00:00 discovering original posts for: https://github.com/DataDog/dd-trace-go/pull/2537
I 2024-02-01 23:57:17.437466+00:00 requests.head https://github.com/DataDog/dd-trace-go/labels/appsec {'headers': {}}
I 2024-02-01 23:57:18.043868+00:00 Received 404
W 2024-02-01 23:57:18.044153+00:00 Couldn't resolve URL https://github.com/DataDog/dd-trace-go/labels/appsec: https://github.com/DataDog/dd-trace-go/labels/appsec
I 2024-02-01 23:57:18.044405+00:00 requests.head https://github.com/DataDog/dd-trace-go/labels/ci {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:18.393453+00:00 Received 404
W 2024-02-01 23:57:18.393718+00:00 Couldn't resolve URL https://github.com/DataDog/dd-trace-go/labels/ci: https://github.com/DataDog/dd-trace-go/labels/ci
I 2024-02-01 23:57:18.393956+00:00 requests.head https://github.com/DataDog/system-tests/ {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:18.822035+00:00 Received 200
I 2024-02-01 23:57:18.822447+00:00 Original post discovery found original posts set(), mentions {'https://github.com/DataDog/system-tests/', 'https://github.com/DataDog/dd-trace-go/labels/ci', 'https://github.com/DataDog/dd-trace-go/labels/appsec'}
I 2024-02-01 23:57:18.823264+00:00 requests.head https://github.com/DataDog/dd-trace-go/pull/2537 {'headers': None}
I 2024-02-01 23:57:19.454818+00:00 Received 200
I 2024-02-01 23:57:19.455163+00:00 starting posse post discovery with syndicated https://github.com/DataDog/dd-trace-go/pull/2537
I 2024-02-01 23:57:19.524303+00:00 requests.head https://dario.im/ {'headers': {}}
I 2024-02-01 23:57:20.318954+00:00 Redirected to https://dario.cat/
I 2024-02-01 23:57:20.319374+00:00 Received 200
D 2024-02-01 23:57:20.319524+00:00 Resolved https://dario.im/ to https://dario.cat/
D 2024-02-01 23:57:20.320552+00:00 fetching author url https://dario.cat/
I 2024-02-01 23:57:20.320710+00:00 requests.get https://dario.cat/ {'gateway': False, 'headers': {}}
I 2024-02-01 23:57:20.622699+00:00 Received 200
D 2024-02-01 23:57:20.630999+00:00 No h-feed found, fallback to top-level h-entrys.
D 2024-02-01 23:57:20.631896+00:00 posse post discovery found no relationship for https://github.com/DataDog/dd-trace-go/pull/2537
D 2024-02-01 23:57:20.860737+00:00 discovering original posts for: https://github.com/DataDog/dd-trace-go/pull/2502
I 2024-02-01 23:57:20.861668+00:00 requests.head https://github.com/DataDog/dd-trace-go/pull/2396/files#diff-a8dc5a7b943aca594d89dc56beced16d9dbc951c6aa0c0ce2d328432dab92c80 {'headers': {}}
I 2024-02-01 23:57:21.658894+00:00 Received 200
I 2024-02-01 23:57:21.659248+00:00 requests.head https://github.com/DataDog/dd-trace-go/assets/62392438/c9c4ed00-a35c-4e10-8054-40ca40df6f01 {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:22.255700+00:00 Redirected to https://github-production-user-asset-6210df.s3.amazonaws.com/62392438/301624426-c9c4ed00-a35c-4e10-8054-40ca40df6f01.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240201T235721Z&X-Amz-Expires=300&X-Amz-Signature=...&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=67680905
I 2024-02-01 23:57:22.256218+00:00 Received 403
W 2024-02-01 23:57:22.256409+00:00 Couldn't resolve URL https://github.com/DataDog/dd-trace-go/assets/62392438/c9c4ed00-a35c-4e10-8054-40ca40df6f01: https://github-production-user-asset-6210df.s3.amazonaws.com/62392438/301624426-c9c4ed00-a35c-4e10-8054-40ca40df6f01.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240201T235721Z&X-Amz-Expires=300&X-Amz-Signature=...&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=67680905
I 2024-02-01 23:57:22.256876+00:00 Original post discovery found original posts set(), mentions {'https://github.com/DataDog/dd-trace-go/pull/2396/files#diff-a8dc5a7b943aca594d89dc56beced16d9dbc951c6aa0c0ce2d328432dab92c80', 'https://github.com/DataDog/system-tests/', 'https://github-production-user-asset-6210df.s3.amazonaws.com/62392438/301624426-c9c4ed00-a35c-4e10-8054-40ca40df6f01.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240201T235721Z&X-Amz-Expires=300&X-Amz-Signature=...&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=67680905'}
I 2024-02-01 23:57:22.257977+00:00 requests.head https://github.com/DataDog/dd-trace-go/pull/2502 {'headers': None}
I 2024-02-01 23:57:24.492582+00:00 Received 200
I 2024-02-01 23:57:24.492915+00:00 starting posse post discovery with syndicated https://github.com/DataDog/dd-trace-go/pull/2502
D 2024-02-01 23:57:24.684052+00:00 activity[content] ## Benchmarks
Benchmark execution time: 2024-01-30 23:23:06
Comparing candidate commit 32b50fbc6fde51abf022da5aaabce0d737fba854 in PR branch `munir/implement-span-links-poc` with baseline commit 8c4834f24951876c7f50a98fa640a96541cc3c04 in branch `main`.
Found 0 performance improvements and 14 performance regressions! Performance is the same for 25 metrics, 2 unstable metrics.
#### scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24
- ๐ฅ `allocated_mem` [+73 bytes; +83 bytes] or [+2.039%; +2.315%]
#### scenario:BenchmarkOTelApiWithCustomTags/otel_api-24
- ๐ฅ `allocated_mem` [+136 bytes; +149 bytes] or [+2.792%; +3.059%]
- ๐ฅ `allocations` [+2; +2] or [+4.545%; +4.545%]
#### scenario:BenchmarkPartialFlushing/Disabled-24
- ๐ฅ `allocated_mem` [+9.135MB; +9.354MB] or [+3.010%; +3.083%]
- ๐ฅ `execution_time` [+6.775ms; +12.972ms] or [+2.402%; +4.599%]
#### scenario:BenchmarkPartialFlushing/Enabled-24
- ๐ฅ `allocated_mem` [+9.276MB; +12.925MB] or [+2.402%; +3.347%]
#### scenario:BenchmarkSingleSpanRetention/no-rules-24
- ๐ฅ `allocated_mem` [+4.848KB; +4.849KB] or [+3.087%; +3.088%]
- ๐ฅ `execution_time` [+9.842ยตs; +10.681ยตs] or [+3.967%; +4.306%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24
- ๐ฅ `allocated_mem` [+4.684KB; +4.857KB] or [+2.937%; +3.045%]
- ๐ฅ `execution_time` [+9.284ยตs; +10.797ยตs] or [+3.712%; +4.317%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24
- ๐ฅ `allocated_mem` [+4.629KB; +4.754KB] or [+2.902%; +2.981%]
- ๐ฅ `execution_time` [+9.046ยตs; +10.539ยตs] or [+3.615%; +4.212%]
#### scenario:BenchmarkStartSpan-24
- ๐ฅ `allocated_mem` [+47 bytes; +48 bytes] or [+3.058%; +3.107%]
#### scenario:BenchmarkTracerAddSpans-24
- ๐ฅ `allocated_mem` [+48 bytes; +49 bytes] or [+2.177%; +2.228%]
=> ## Benchmarks
Benchmark execution time: 2024-02-01 19:10:30
Comparing candidate commit c75a0953c52224c41f53341b9acd90e58aef09d8 in PR branch `munir/implement-span-links-poc` with baseline commit e707cc94fdd4e663fccdb6ab3fb3258539444bc0 in branch `main`.
Found 0 performance improvements and 12 performance regressions! Performance is the same for 27 metrics, 2 unstable metrics.
#### scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24
- ๐ฅ `allocated_mem` [+76 bytes; +84 bytes] or [+2.101%; +2.326%]
#### scenario:BenchmarkPartialFlushing/Disabled-24
- ๐ฅ `allocated_mem` [+9.143MB; +9.430MB] or [+3.014%; +3.108%]
#### scenario:BenchmarkPartialFlushing/Enabled-24
- ๐ฅ `allocated_mem` [+9.313MB; +13.147MB] or [+2.411%; +3.404%]
#### scenario:BenchmarkSingleSpanRetention/no-rules-24
- ๐ฅ `allocated_mem` [+4.848KB; +4.849KB] or [+3.087%; +3.087%]
- ๐ฅ `execution_time` [+9.682ยตs; +11.064ยตs] or [+3.901%; +4.457%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24
- ๐ฅ `allocated_mem` [+4.732KB; +4.906KB] or [+2.968%; +3.077%]
- ๐ฅ `execution_time` [+9.372ยตs; +10.696ยตs] or [+3.745%; +4.274%]
#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24
- ๐ฅ `allocated_mem` [+4.700KB; +4.895KB] or [+2.948%; +3.071%]
- ๐ฅ `execution_time` [+8.437ยตs; +10.194ยตs] or [+3.364%; +4.064%]
#### scenario:BenchmarkStartSpan-24
- ๐ฅ `allocated_mem` [+48 bytes; +48 bytes] or [+3.076%; +3.128%]
#### scenario:BenchmarkTracerAddSpans-24
- ๐ฅ `allocated_mem` [+49 bytes; +49 bytes] or [+2.212%; +2.212%]
- ๐ฅ `execution_time` [+85.364ns; +131.036ns] or [+2.110%; +3.239%]
I 2024-02-01 23:57:24.684849+00:00 Response changed! Re-propagating. Original: Response(key=Key('Response', 'tag:github.com,2013:DataDog:dd-trace-go:IC_kwDOBAi6ic5whxJ7'), activities_json=['{"id":"tag:github.com,2013:DataDog:dd-trace-go:2502","url":"https://github.com/DataDog/dd-trace-go/pull/2502","content":"### What does this PR do?\\n\\nImplements Span Links in the opentelemetry datadog API\\n\\nBuilds on: https://github.com/DataDog/dd-trace-go/pull/2396/files#diff-a8dc5a7b943aca594d89dc56beced16d9dbc951c6aa0c0ce2d328432dab92c80.\\n\\n### Motivation\\n\\nPreviously setting a span link on an OpenTelemetry span would result in a no-op with this change SpanLinks will be serialized and sent to the Datadog Agent.\\n\\n### Reviewer\'s Checklist\\n<!--\\n* Authors can use this list as a reference to ensure that there are no problems\\n during the review but the signing off is to be done by the reviewer(s).\\n-->\\n\\n- [ ] Changed code ... unit tests for its functionality at or near 100% coverage.\\n- [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag.\\n- [ ] There is a benchmark for any new code, ... changes to existing code.\\n- [ ] If this interacts with the agent in a new way, a system test has been added.\\n- [ ] Add an appropriate team label so this PR gets put in the right place for the release notes.\\n- [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.\\n\\nFor Datadog employees:\\n\\n- [ ] If this PR touches code ... handles credentials of any kind, such as Datadog API keys, I\'ve requested a review from `@DataDog/security-design-and-guidance`.\\n- [ ] This PR doesn\'t touch any of that.\\n\\nUnsure? Have a question? Request a review!\\n"}'], created=DatetimeWithNanoseconds(2024, 1, 11, 21, 35, 10, 991957, tzinfo=datetime.timezone.utc), error=[], failed=[], leased_until=DatetimeWithNanoseconds(2024, 1, 31, 22, 59, 56, 183675, tzinfo=datetime.timezone.utc), old_response_jsons=['{"id":"tag:github.com,2013:DataDog:dd-trace-go:IC_kwDOBAi6ic5whxJ7","url":"https://github.com/DataDog/dd-trace-go/pull/2502#issuecomment-1887900283","author":{"id":"tag:github.com,2013:MDM6Qm90MzQ1NzgxMTY=","url":"https://github.com/apps/pr-commenter","displayName":"pr-commenter[bot]","objectType":"person","username":"pr-commenter[bot]","image":{"url":"https://avatars.githubusercontent.com/u/365230?v=4"}},"content":"## Benchmarks\\n\\nBenchmark execution time: 2024-01-30 18:02:18\\n\\nComparing candidate commit bc0cb2e8c9f0e894462fbcd506e25c6be8bc594f in PR branch `munir/implement-span-links-poc` with baseline commit c527cccd7386a26ad6144d92f2d741277d570159 in branch `main`.\\n\\nFound 0 performance improvements and 10 performance regressions! Performance is the same for 30 metrics, 1 unstable metrics.\\n\\n#### scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+76 bytes; +83 bytes] or [+2.120%; +2.296%]\\n\\n#### scenario:BenchmarkOTelApiWithCustomTags/otel_api-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+138 bytes; +149 bytes] or [+2.848%; +3.073%]\\n- \\ud83d\\udfe5 `allocations` [+2; +2] or [+4.545%; +4.545%]\\n\\n#### scenario:BenchmarkPartialFlushing/Disabled-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+9.132MB; +9.347MB] or [+3.010%; +3.080%]\\n\\n#### scenario:BenchmarkPartialFlushing/Enabled-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+9.765MB; +14.312MB] or [+2.536%; +3.717%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/no-rules-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.847KB; +4.849KB] or [+3.087%; +3.087%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.748KB; +4.948KB] or [+2.978%; +3.104%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.752KB; +4.924KB] or [+2.981%; +3.089%]\\n\\n#### scenario:BenchmarkStartSpan-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+47 bytes; +48 bytes] or [+3.044%; +3.095%]\\n\\n#### scenario:BenchmarkTracerAddSpans-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+48 bytes; +48 bytes] or [+2.178%; +2.209%]\\n\\n\\n\\n","published":"2024-01-11T20:18:02+00:00","updated":"2024-01-30T18:07:34+00:00","objectType":"comment","inReplyTo":[{"url":"https://github.com/DataDog/dd-trace-go/pull/2502"}]}', '{"id":"tag:github.com,2013:DataDog:dd-trace-go:IC_kwDOBAi6ic5whxJ7","url":"https://github.com/DataDog/dd-trace-go/pull/2502#issuecomment-1887900283","author":{"id":"tag:github.com,2013:MDM6Qm90MzQ1NzgxMTY=","url":"https://github.com/apps/pr-commenter","objectType":"person","displayName":"pr-commenter[bot]","username":"pr-commenter[bot]","image":{"url":"https://avatars.githubusercontent.com/u/365230?v=4"}},"content":"## Benchmarks\\n\\nBenchmark execution time: 2024-01-11 20:12:35\\n\\nComparing candidate commit 01edc9ac1b551d9b51d53ad23c08a7d11eb061f7 in PR branch `munir/implement-span-links-poc` with baseline commit 249640f18ad6cfea81b0a155920f14dc9a8348b0 in branch `main`.\\n\\nFound 0 performance improvements and 7 performance regressions! Performance is the same for 32 metrics, 2 unstable metrics.\\n\\n#### scenario:BenchmarkPartialFlushing/Disabled-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+7.791MB; +8.071MB] or [+2.568%; +2.660%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/no-rules-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.847KB; +4.848KB] or [+3.087%; +3.087%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.761KB; +4.899KB] or [+2.986%; +3.072%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.829KB; +5.003KB] or [+3.030%; +3.139%]\\n\\n#### scenario:BenchmarkStartSpan-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+47 bytes; +48 bytes] or [+3.068%; +3.109%]\\n\\n#### scenario:BenchmarkTracerAddSpans-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+48 bytes; +49 bytes] or [+2.199%; +2.226%]\\n- \\ud83d\\udfe5 `execution_time` [+88.687ns; +131.513ns] or [+2.194%; +3.254%]\\n\\n\\n\\n","published":"2024-01-11T20:18:02+00:00","updated":"2024-01-11T20:18:02+00:00","objectType":"comment","inReplyTo":[{"url":"https://github.com/DataDog/dd-trace-go/pull/2502"}]}'], original_posts=[], response_json='{"id":"tag:github.com,2013:DataDog:dd-trace-go:IC_kwDOBAi6ic5whxJ7","url":"https://github.com/DataDog/dd-trace-go/pull/2502#issuecomment-1887900283","author":{"id":"tag:github.com,2013:MDM6Qm90MzQ1NzgxMTY=","url":"https://github.com/apps/pr-commenter","displayName":"pr-commenter[bot]","objectType":"person","username":"pr-commenter[bot]","image":{"url":"https://avatars.githubusercontent.com/u/365230?v=4"}},"content":"## Benchmarks\\n\\nBenchmark execution time: 2024-01-30 23:23:06\\n\\nComparing candidate commit 32b50fbc6fde51abf022da5aaabce0d737fba854 in PR branch `munir/implement-span-links-poc` with baseline commit 8c4834f24951876c7f50a98fa640a96541cc3c04 in branch `main`.\\n\\nFound 0 performance improvements and 14 performance regressions! Performance is the same for 25 metrics, 2 unstable metrics.\\n\\n#### scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+73 bytes; +83 bytes] or [+2.039%; +2.315%]\\n\\n#### scenario:BenchmarkOTelApiWithCustomTags/otel_api-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+136 bytes; +149 bytes] or [+2.792%; +3.059%]\\n- \\ud83d\\udfe5 `allocations` [+2; +2] or [+4.545%; +4.545%]\\n\\n#### scenario:BenchmarkPartialFlushing/Disabled-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+9.135MB; +9.354MB] or [+3.010%; +3.083%]\\n- \\ud83d\\udfe5 `execution_time` [+6.775ms; +12.972ms] or [+2.402%; +4.599%]\\n\\n#### scenario:BenchmarkPartialFlushing/Enabled-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+9.276MB; +12.925MB] or [+2.402%; +3.347%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/no-rules-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.848KB; +4.849KB] or [+3.087%; +3.088%]\\n- \\ud83d\\udfe5 `execution_time` [+9.842\\u00b5s; +10.681\\u00b5s] or [+3.967%; +4.306%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.684KB; +4.857KB] or [+2.937%; +3.045%]\\n- \\ud83d\\udfe5 `execution_time` [+9.284\\u00b5s; +10.797\\u00b5s] or [+3.712%; +4.317%]\\n\\n#### scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+4.629KB; +4.754KB] or [+2.902%; +2.981%]\\n- \\ud83d\\udfe5 `execution_time` [+9.046\\u00b5s; +10.539\\u00b5s] or [+3.615%; +4.212%]\\n\\n#### scenario:BenchmarkStartSpan-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+47 bytes; +48 bytes] or [+3.058%; +3.107%]\\n\\n#### scenario:BenchmarkTracerAddSpans-24\\n\\n- \\ud83d\\udfe5 `allocated_mem` [+48 bytes; +49 bytes] or [+2.177%; +2.228%]\\n\\n\\n\\n","published":"2024-01-11T20:18:02+00:00","updated":"2024-01-30T23:28:29+00:00","objectType":"comment","inReplyTo":[{"url":"https://github.com/DataDog/dd-trace-go/pull/2502"}]}', sent=[], skipped=[], source=Key('GitHub', 'imdario'), status='complete', type='comment', unsent=[], updated=datetime.datetime(2024, 2, 1, 23, 57, 24, 632663, tzinfo=datetime.timezone.utc), urls_to_activity=None)
I 2024-02-01 23:57:24.813557+00:00 Added propagate task projects/brid-gy/locations/us-central1/queues/propagate/tasks/7821947722893702756 with ETA None: {'app_engine_http_request': {'http_method': 'POST', 'relative_uri': '/_ah/queue/propagate', 'body': b'response_key=agdicmlkLWd5ckkLEghSZXNwb25zZSI7dGFnOmdpdGh1Yi5jb20sMjAxMzpEYXRhRG9nOmRkLXRyYWNlLWdvOklDX2t3RE9CQWk2aWM1d2h4SjcM', 'headers': {'Content-Type': 'application/x-www-form-urlencoded'}}}
D 2024-02-01 23:57:24.813873+00:00 discovering original posts for: https://github.com/DataDog/datadog-agent/pull/22491
I 2024-02-01 23:57:24.814791+00:00 requests.head https://github.com/DataDog/datadog-agent/labels/changelog/no-changelog {'headers': {}}
I 2024-02-01 23:57:25.670960+00:00 Received 404
W 2024-02-01 23:57:25.671230+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/labels/changelog/no-changelog: https://github.com/DataDog/datadog-agent/labels/changelog/no-changelog
I 2024-02-01 23:57:25.671516+00:00 requests.head https://github.com/DataDog/datadog-agent/labels/qa/skip-qa {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:26.576954+00:00 Received 404
W 2024-02-01 23:57:26.577218+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/labels/qa/skip-qa: https://github.com/DataDog/datadog-agent/labels/qa/skip-qa
I 2024-02-01 23:57:26.577439+00:00 requests.head https://github.com/DataDog/datadog-agent/labels/mergequeue-status%3A%20done {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:26.914263+00:00 Received 404
W 2024-02-01 23:57:26.914951+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/labels/mergequeue-status%3A%20done: https://github.com/DataDog/datadog-agent/labels/mergequeue-status%3A%20done
I 2024-02-01 23:57:26.915241+00:00 requests.head https://github.com/DataDog/datadog-agent/labels/qa/no-code-change {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:28.157301+00:00 Received 404
W 2024-02-01 23:57:28.157596+00:00 Couldn't resolve URL https://github.com/DataDog/datadog-agent/labels/qa/no-code-change: https://github.com/DataDog/datadog-agent/labels/qa/no-code-change
I 2024-02-01 23:57:28.157859+00:00 requests.head https://github.com/DataDog/datadog-agent/pull/22373 {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:29.106394+00:00 Received 200
I 2024-02-01 23:57:29.107330+00:00 requests.head https://github.com/DataDog/datadog-agent/pull/22495 {'headers': {'User-Agent': '...'}}
I 2024-02-01 23:57:29.972496+00:00 Received 200
I 2024-02-01 23:57:29.973067+00:00 Original post discovery found original posts set(), mentions {'https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FDataDog%2Fdocumentation%2Fissues%2Fnew', 'https://github.com/DataDog/datadog-agent/blob/main/docs/dev/contributing.md#reno', 'https://github.com/DataDog/datadog-agent/pull/22495', 'https://github.com/DataDog/datadog-agent/labels/changelog/no-changelog', 'https://github.com/DataDog/datadog-agent/labels/qa/skip-qa', 'https://github.com/DataDog/datadog-agent/labels/team/agent-apm', 'https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml', 'https://github.com/DataDog/datadog-agent/pull/22373', 'https://github.com/DataDog/datadog-agent/labels/qa/no-code-change', 'https://github.com/DataDog/datadog-agent/labels/mergequeue-status%3A%20done'}
I 2024-02-01 23:57:29.974217+00:00 requests.head https://github.com/DataDog/datadog-agent/pull/22491 {'headers': None}
I 2024-02-01 23:57:30.909744+00:00 Received 200
I 2024-02-01 23:57:30.910113+00:00 starting posse post discovery with syndicated https://github.com/DataDog/datadog-agent/pull/22491
D 2024-02-01 23:57:30.990083+00:00 skipping https://dario.im/, already fetched this round
D 2024-02-01 23:57:30.994754+00:00 posse post discovery found no relationship for https://github.com/DataDog/datadog-agent/pull/22491
I 2024-02-01 23:57:32.794203+00:00 skipping refetch h-feed. last-syndication-url None, last-refetch 2018-04-05 14:35:37.888000+00:00
I 2024-02-01 23:57:32.811886+00:00 Updating imdario (GitHub) /github/imdario : {'recent_private_posts': 0, 'last_polled': datetime.datetime(2024, 2, 1, 23, 56, 53, 202973, tzinfo=datetime.timezone.utc), 'poll_status': 'ok', 'last_activities_etag': 'Thu, 01 Feb 2024 21:38:41 GMT'}
I 2024-02-01 23:57:32.917441+00:00 Added poll task projects/brid-gy/locations/us-central1/queues/poll/tasks/8856345401099413276 with ETA 1706934315: {'app_engine_http_request': {'http_method': 'POST', 'relative_uri': '/_ah/queue/poll', 'body': b'source_key=agdicmlkLWd5chMLEgZHaXRIdWIiB2ltZGFyaW8M&last_polled=2024-02-01-23-56-53', 'headers': {'Content-Type': 'application/x-www-form-urlencoded'}}, 'schedule_time': seconds: 1706934315
}