Changes

Summary

1. #1703 fixed wrongly emptied out search index on policy retrieval error due to internal timeouts (details)
2. update maven plugins to latest versions (details)
3. #1739 provide the traceparent header as MDC value in logs (details)
4. add IntLevelJsonProvider as logstash/logback provider, adding the log level as "intLevel" (details)
5. #1739 log 2 separate MDC fields: "traceparent-trace-id" and "traceparent-span-id" (details)
6. #1739 removed IntLevelJsonProvider from default logback files in Helm chart (details)
7. #1739 undid chart changes (details)
8. downgrade to Scala 2.13.10 due to unit test errors in ThingsSseRouteBuilderTest (details)
9. update to Scala 2.13.12 again and fix root cause, using `CompletableFuture.completedStage` instead of `CompletableFuture.completedFuture` (details)
10. provide a new function to join placeholder array pipeline elements into a single string (details)
11. update to Swagger-UI 5.7.2 (details)
12. Add support for MQTT message expiry interval (details)
13. Add MQTT version information to MQTT headers (details)
14. Update documentation (details)
15. Add javadoc (details)
16. New Ingress resources only for devops paths (details)
17. Reducing time wait for ingress controller shutdown (details)
18. updating chart version to 3.3.8 and app version to nightly (details)
19. update dependencies (details)
20. Do not allow MQTT 5 user properties to overwrite special headers (details)
21. Do not include known MQTT headers in outgoing message based on feature toggle (details)
22. fix(helm): fix missing whitespace in JAVA_TOOL_OPTIONS (details)
23. Implement benchmark-tool, based on k6-runner (details)
24. renaming akkaClusterMaxFramesize in Helm config after migration to Pekko (details)
25. use 3.4.0-M1 milestone as appVersion instead of "nightly" (details)
26. Add escapes for placeholders, fix urls, add dependency versions, add myself as author (details)
27. improved logging in ConnectionIdsRetrievalActor (details)
28. Revert "improved logging in ConnectionIdsRetrievalActor" (details)
29. improved logging in ConnectionIdsRetrievalActor (details)
30. improved logging in ConnectionIdsRetrievalActor, pt.2 (details)
31. further stabilize connectionIds retrieval (details)
32. Rename post filename to match the permalink (details)
33. fixed ConnectionIdsRetrievalActorTest (details)
34. reduced Kamon startup warnings and logs (details)
35. update to Kamon 2.6.5 (details)
36. prepare Ditto 3.4.0 release notes (details)
37. reduce log level of 2 WARN log messages to INFO (details)
38. several small Ditto UI improvements (details)
39. made Ditto UI more responsive for mobile layout (details)
40. prevent browser from suggesting stored credentials for "search" input field (details)
41. increase size of "Things" editor when editing a thing (details)
42. increase width of edit mode editors to 90% (details)
43. apply fixes regarding message templates (details)
44. show "_metadata" in readonly things JSON viewer (details)
45. #1712 fix patching a thing, removing items did not correctly upgrade the thing on the UI (details)
46. added documentation for SSE "_context" extraField (details)
47. fixed NormalizedMessageMapperTest, including new "value" field to expectations (details)
48. #1631 fix putting metadata on a single property did not work (details)
49. fix SSE overwriting message input fields (details)
50. maintain SSE "Fields" determining behavior in backward compatible way (details)
51. enhance release notes with UI enhancements (details)
52. preserve httpStatus for other reason on faild create of thing (details)
53. provide new option for "if-equal" header: "skip-minimizing-merge" (details)
54. added unit test for "if-equal" header: "skip-minimizing-merge" (details)
55. update Ditto 3.4.0 publication date (details)
56. update appVersion of Helm Chart to 3.4.0 (details)
57. increase binary-compatibility-check.version to 3.4.0 after release (details)
58. disable reduced pom generation (details)
59. #1780 fix update headers of ConnectionClosed (details)
60. remove `_context` from thingEditor upon changes (details)
61. fix that ThingFieldSelector did not allow specific paths of "_metadata" (details)
62. bump swagger-ui version to resolve security issues (details)
63. fix swaggerui-init container script (details)
64. Bump com.rabbitmq:amqp-client from 5.14.3 to 5.18.0 in /bom (details)
65. Move x-Forwarded-User and x-ditto-pre-authenticated from ingress-controller to values.yaml config (details)
66. #1748 Add honoTenantId configuration for HonoConnection. (details)
67. Bump chart version (details)
68. Bump Chart-Version to 3.4.2 (details)
69. fix preserving DittoHeaders when encountering a JsonParseException (details)
70. bump helm/chart-testing-action to version 2.6.1 (details)
71. Update .github/workflows/helm-chart.yml (details)
72. Update .github/workflows/helm-chart.yml (details)
73. fix regression introduced in #1792 (details)
74. UI: fix that sending message payload `false` did not work in UI "Send Message" (details)
75. Add piggyback commands to ui (details)
76. Move try catch block to correct place, make sure onRequestDone is called in case of callDittoREST method error thrown (details)
77. Move stringifyPretty to utils, replace all JSON.stringify usage with util function. (details)
78. Moved piggyback to operations tab (details)
79. add newlines at end of files (details)
80. move piggyback files to operations/, rename operations to serviceLogging (details)
81. Fix path for piggyback request, remove commands not part of ditto (details)
82. fix that JsonObjectBuilder.remove removes too much in certain cases (details)
83. fix search query mutiple sort option (details)
84. provide Ditto 3.4.1 release notes (details)
85. adjust Helm versions for 3.4.1 release (details)
86. fix release date in 3.4.1 release notes (details)
87. #1801 Add trusted certificates configuration for HonoConnection. (details)
88. Remove util method for set editor value, replace calls with setEditorValue(value, -1) (details)
89. UI - Allow editors in full screen mode (details)
90. UI - enable scrolling back if not fullscreen (details)
91. Fix set command from template to command editor (details)
92. setValue instead of setEditorValue (details)
93. Add integration tests for MQTT (details)
94. Update MqttClientActorIT to test MQTT 3 and 5 (details)
95. Refactor (details)
96. Use rule instead of before for disabling tracing (details)
97. Start mosquitto container via tests (details)
98. Small improvements to tests (details)
99. Run MqttClientActorIT tests for clean session as well (details)
100. Small fixes (details)
101. Fix year in file headers (details)
102. Rework consuming messages from MQTT broker (details)
103. Minor changes (details)
104. Add some checks to BufferingFlowableWrapper (details)
105. Lower visibility of GenericMqttConsumingClient (details)
106. Uncomment assertion that all messages aare acked in testMultipleSources (details)
107. UI - Support for policy imports (details)
108. UI - policy imports review (details)
109. UI - policy imports review 2 (details)
110. UI - Send messages improvements (details)
111. UI - policy imports missing headers in new files (details)
112. UI - bug fix for things without features (details)
113. fix that non-successful message responses were not displayed in ACE editor (details)
114. display attributes in UI in an aceEditor instead of a simple inputfield (details)
115. enhance "Incoming Thing Updates" in UI to provide more information at a glance (details)
116. fix that a "wrong" JsonPointer in a search RQL query lead to ERRORs in Ditto (details)
117. address review finding, adding class resizable_pane and height (details)
118. improved error toast and showing errors from Ditto (details)
119. UI - fix operations new empty logger (details)
120. UI - fix 401 on startup (details)
121. preserve maxPid in listNewestActiveSnapshotsByBatch aggregation (details)
122. removed X-Original-URI (details)
123. add "final" to BufferingFlowableWrapper class (details)
124. #1826 fix JsonFieldSelectorTrie logic for objects and parts of those objects both being included in selected fields (details)
125. Added 50m log limit to all docker containers (details)
126. #1806 make a common metric prefix configurable for all gathered Ditto custom metrics (details)
127. #1806 provide configuration for operator defined custom metrics (details)
128. #1806 provide Helm configuration for metric prefix and custom metrics (details)
129. #1806 provide documentation (details)
130. #1806 added missing param javadoc (details)
131. #1806 bumped chart version to pass linting (details)
132. migrate logging to SLF4J 2.0.x and Logback 1.3.x (details)
133. provide Ditto 3.4.2 release notes (details)
134. prepare helm Chart.yaml for 3.4.2 (details)
135. fixed unit tests relying on old SLF4J "Binder" mechanism (details)
136. use logback 1.4.13 instead of 1.3.13 (details)
137. updated other logging related dependency versions (details)
138. #1883 return pre-authenticated configuration to location-snippet (details)
139. Updating chart version (details)
140. update logback 1.2 to mitigate CVE (details)
141. prepare release notes for Ditto 3.4.3 (details)
142. exclude false positive method from binary comp. check (details)
143. #1699 made building the Ditto UI image optional and configurable in build-images.sh (details)
144. fixed "deleted" action column values containing all features (details)
145. #1844 fix resolving revision or timestamp from the future was not directly denied (details)
146. #1844 fix wrong "reduction" when calculating retrieving `at-historical-revision` (details)
147. enhance WoT skeleton creation with more options (details)
148. remove noisy INFO log statement in StreamingSessionActor (details)
149. fixed that "condition" query param could not be provided as form field (details)
150. enhanced documentation, adding the "condition" for POST search (details)
151. updated docker-compose files to version 3.4, fixing broken file due to "logging" only being available in version "3" (details)
152. Fix maxWait value to match the newest version of xk6-kafka (details)
153. Remove test-local.env, rename test-cluster.env to test.env, add copyright headers to ditto-helm-values.yaml (details)
154. prepare Ditto 3.4.4 release notes (details)
155. bump Helm chart versions to 3.4.4 (details)
156. added license headers for 2024 (details)
157. Declare connectionStatusError variable (details)
158. #1854 enhance time:now* placeholders to calculate plus and minus from now (details)
159. #1854 provide documentation about enhanced now placeholder functionality (details)
160. #1583 apply RQL based filtering when streaming "historical" thing events (details)
161. #1583 provided documentation for filtering in historical thing events (details)
162. Provide same behavior for filtering in arrays via search's "filter" for specified "condition" (details)
163. UI - add filter for tables (details)
164. fix _policy enrichment on ThingCreated event (details)
165. update EnforcementFlowTest tests to supports updated cache retrieval for _policy enrichment (details)
166. fix license header (details)
167. added missing `logstash-logback-encoder.version` property (details)
168. fixed logging testing (details)
169. Enhance things search slot by displaying the amount of matching things (details)
170. Provide option to provide negative numbers to historical event streaming (details)
171. fixed copy&paste errors for calling wrapped `containsFlatteningArrays` (details)
172. Add subsystem health check to liveness checks (details)
173. Ditto 1521: Initial submission of scoping search fields for things. (details)
174. Corrected license headers. (details)
175. More corrected license headers. (details)
176. bumped slf4j and logback versions to latest patch releases (details)
177. 1. Corrected issues based on review.  2.  Added documentation on this feature. (details)
178. #1869 use cache in order to load imported policies when policies are loaded after invalidation in search (details)
179. Replaced namespace string with pattern. (details)
180. Added a namespace-pattern configuration. (details)
181. review commit: (details)
182. provided Helm configuration for configuring indexedFields for namespaces (details)
183. #1869 provided missing javadocs, fixed test (details)
184. provided unit test for testing an imported policy is only loaded once by using cache (details)
185. fixed trying to process empty "currentWriteModel" leading to BsonException (details)
186. #1839: introduce script to Helm chart updating pod-deletion-cost (details)
187. Fix helm-chart securityContext identation if openshift enabled (details)
188. Bump chart version (details)
189. prevent browser autocomplete to suggest credentials for several input fields (details)
190. bump chart version, fix indentation (details)

#1703 fixed wrongly emptied out search index on policy retrieval error due to internal timeouts

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

update maven plugins to latest versions

* and update legal / dependencies documents accordingly

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

#1739 provide the traceparent header as MDC value in logs

* generify the fields to provide to the MDC in CommonMdcEntryKey enum
* rename "x-correlation-id" in logs to just "correlation-id"
* exchange some places where only correlationId was extracted from a map of headers with parsing all the headers for MDC worthy fields

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

add IntLevelJsonProvider as logstash/logback provider, adding the log level as "intLevel"

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

#1739 log 2 separate MDC fields: "traceparent-trace-id" and "traceparent-span-id"

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

#1739 removed IntLevelJsonProvider from default logback files in Helm chart

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

#1739 undid chart changes

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

downgrade to Scala 2.13.10 due to unit test errors in ThingsSseRouteBuilderTest

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

update to Scala 2.13.12 again and fix root cause, using `CompletableFuture.completedStage` instead of `CompletableFuture.completedFuture`

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

provide a new function to join placeholder array pipeline elements into a single string

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

update to Swagger-UI 5.7.2

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

Add support for MQTT message expiry interval

Signed-off-by: Dmitriy Barbul <dimabarbul@gmail.com>

Add MQTT version information to MQTT headers

Signed-off-by: Dmitriy Barbul <dimabarbul@gmail.com>

Update documentation

Signed-off-by: Dmitriy Barbul <dimabarbul@gmail.com>

Add javadoc

New Ingress resources only for devops paths

This specialized Ingress resource is to allow basic auth of devops path
with `devops` and `status` users through Ingress Controller.
Previously these paths were assigned to api Ingress, and they required
ditto user credentials.

Reducing time wait for ingress controller shutdown

According to https://medium.com/codecademy-engineering/kubernetes-nginx-and-zero-downtime-in-production-2c910c6a5ed8 95 seconds seems like too much for ingress controller preStop shutdown.

updating chart version to 3.3.8 and app version to nightly

update dependencies

* Kafka client to 3.5.1
* HiveMQ MQTT client to 1.3.3
* QPID JMS client to 1.10.0
* AMQP-Client to 5.14.3
* Netty to 4.1.99
* Logback to 1.2.12
* Janino to 3.1.10
* Kamon to 2.6.4
* Caffeine to 3.1.8
* Classindex to 3.13
* Some test dependencies to latest versions

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

Do not allow MQTT 5 user properties to overwrite special headers

Do not include known MQTT headers in outgoing message based on feature toggle

fix(helm): fix missing whitespace in JAVA_TOOL_OPTIONS

Signed-off-by: Mathias Maes <mathias.maes@aloxy.io>

Implement benchmark-tool, based on k6-runner

Signed-off-by: Vasil Vasilev <vasil.vasilev@bosch.com>

renaming akkaClusterMaxFramesize in Helm config after migration to Pekko

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

use 3.4.0-M1 milestone as appVersion instead of "nightly"

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

Add escapes for placeholders, fix urls, add dependency versions, add myself as author

Signed-off-by: Vasil Vasilev <vasil.vasilev@bosch.com>

improved logging in ConnectionIdsRetrievalActor

* got rid of obsolete AutoDiscardingDiagnosticLoggingAdapter.java

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

Revert "improved logging in ConnectionIdsRetrievalActor"

This reverts commit 120a860a770280c1a30ca52ec0f503afa2594432.

improved logging in ConnectionIdsRetrievalActor

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

improved logging in ConnectionIdsRetrievalActor, pt.2

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

further stabilize connectionIds retrieval

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

Rename post filename to match the permalink

Signed-off-by: Vasil Vasilev <vasil.vasilev@bosch.com>

fixed ConnectionIdsRetrievalActorTest

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

reduced Kamon startup warnings and logs

* made loading of "otel-trace-reporter" configurable and by default, don't load it
* introduce new env var `DITTO_TRACING_OTEL_TRACE_REPORTER_ENABLED` which can in
  addition to `DITTO_TRACING_ENABLED` be used to just switch off OTEL reporting of traces

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

update to Kamon 2.6.5

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

prepare Ditto 3.4.0 release notes

* and update SECURITY.md to support 3.3 and 3.4 with security patches

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

reduce log level of 2 WARN log messages to INFO

* and get rid of thrown IllegalStateException, causing log WARNings

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

several small Ditto UI improvements

* add a tab "Message to Thing" to send thing messages
* add a loading spinner to the "Send" (message) button and deactivate it while sending
* update a complete Thing using "PATCH" and with the new 3.4.0 header "if-equal: skip-minimizing-merge"
* only send eTag if it could be retrieved when updating complete thing
* added missing "ilike" predicate to the search slot

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

made Ditto UI more responsive for mobile layout

* mainly the navigation

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

prevent browser from suggesting stored credentials for "search" input field

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

increase size of "Things" editor when editing a thing

increase width of edit mode editors to 90%

* apply bigger width also for editing feature and editing connection
* capitalize "Send Message"

apply fixes regarding message templates

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

show "_metadata" in readonly things JSON viewer

* in "edit" mode, "_metadata" is removed, so writing metadata is not supported yet

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

#1712 fix patching a thing, removing items did not correctly upgrade the thing on the UI

* fixed by solving the merge on Ditto server side, selecting all "extraFields" for each change .. that way, the complete updated thing is emitted via the SSE instead only the diff
* as a result, also deleted the "merge" functionality of the UI, also removed no-longer needed "lodash" dependency

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

added documentation for SSE "_context" extraField

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

fixed NormalizedMessageMapperTest, including new "value" field to expectations

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

#1631 fix putting metadata on a single property did not work

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

fix SSE overwriting message input fields

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

maintain SSE "Fields" determining behavior in backward compatible way

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

enhance release notes with UI enhancements

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

preserve httpStatus for other reason on faild create of thing

Signed-off-by: Aleksandar Stanchev <aleksandar.stanchev@bosch.com>

provide new option for "if-equal" header: "skip-minimizing-merge"

* which reduces the part of a merge/patch command to the actually changed elements, removing non-changed elements
* this reduces e.g. required storage in the MongoDB by a lot, if redundant data is sent often
* this also reduces the event payload to the actually changed parts of the thing

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

added unit test for "if-equal" header: "skip-minimizing-merge"

* also added documentation
* enhanced release notes + blogpost about release with last-minute addition

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

update Ditto 3.4.0 publication date

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

update appVersion of Helm Chart to 3.4.0

increase binary-compatibility-check.version to 3.4.0 after release

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

disable reduced pom generation

Signed-off-by: Aleksandar Stanchev <aleksandar.stanchev@bosch.com>

#1780 fix update headers of ConnectionClosed

Signed-off-by: Aleksandar Stanchev <aleksandar.stanchev@bosch.com>

remove `_context` from thingEditor upon changes

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

fix that ThingFieldSelector did not allow specific paths of "_metadata"

* it only allowed to select top-level "_metadata" - which does not make sense to limit, as even selecting only specific _metadata is already supported
* also add missing support for featureId wildcard when selecting e.g. "_metadata/features/*/properties"

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

bump swagger-ui version to resolve security issues

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

fix swaggerui-init container script

Bump com.rabbitmq:amqp-client from 5.14.3 to 5.18.0 in /bom

Bumps [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) from 5.14.3 to 5.18.0.
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.14.3...v5.18.0)

---
updated-dependencies:
- dependency-name: com.rabbitmq:amqp-client
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Move x-Forwarded-User and x-ditto-pre-authenticated from ingress-controller to values.yaml config

Signed-off-by: Dominik Mlasko <dominik.mlasko@blum.com>

#1748 Add honoTenantId configuration for HonoConnection.

Bump chart version

Signed-off-by: Dominik Mlasko <dominik.mlasko@blum.com>

Bump Chart-Version to 3.4.2

fix preserving DittoHeaders when encountering a JsonParseException

* e.g. containing correlation-id, trace-context, etc.