diff --git a/docs/develop/dotnet/best-practices/data-handling/data-conversion.mdx b/docs/develop/dotnet/best-practices/data-handling/data-conversion.mdx new file mode 100644 index 0000000000..a91d6b35c4 --- /dev/null +++ b/docs/develop/dotnet/best-practices/data-handling/data-conversion.mdx @@ -0,0 +1,70 @@ +--- +id: data-conversion +title: Payload conversion - .NET SDK +sidebar_label: Payload conversion +toc_max_heading_level: 3 +tags: + - Data Converters + - .NET SDK + - Temporal SDKs +description: Customize how Temporal serializes application objects using Payload Converters in the .NET SDK. +--- + +import { CaptionedImage } from '@site/src/components'; + +## Payload conversion + +Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. + +### Conversion sequence {/* #conversion-sequence */} + +The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. +You can set multiple encoding Payload Converters to run your conversions. +When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. + +Payload Converters can be customized independently of a Payload Codec. +Temporal's Converter architecture looks like this: + + + +## Custom Payload Converter {/* #custom-payload-converter */} + +Data converters are used to convert raw Temporal payloads to/from actual .NET types. +A custom data converter can be set via the `DataConverter` option when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. +Payload converters convert .NET values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures. + +Data converters are in the `Temporalio.Converters` namespace. +The default data converter uses a default payload converter, which supports the following types: + +- `null` +- `byte[]` +- `Google.Protobuf.IMessage` instances +- Anything that `System.Text.Json` supports +- `IRawValue` as unconverted raw payloads + +Custom converters can be created for all uses. For example, to create client with a data converter that converts all C# +property names to camel case, you would: + +```csharp +using System.Text.Json; +using Temporalio.Client; +using Temporalio.Converters; + +public class CamelCasePayloadConverter : DefaultPayloadConverter +{ + public CamelCasePayloadConverter() + : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase }) + { + } +} + +var client = await TemporalClient.ConnectAsync(new() +{ + TargetHost = "localhost:7233", + Namespace = "my-namespace", + DataConverter = DataConverter.Default with { PayloadConverter = new CamelCasePayloadConverter() }, +}); +``` diff --git a/docs/develop/dotnet/best-practices/converters-and-encryption.mdx b/docs/develop/dotnet/best-practices/data-handling/data-encryption.mdx similarity index 62% rename from docs/develop/dotnet/best-practices/converters-and-encryption.mdx rename to docs/develop/dotnet/best-practices/data-handling/data-encryption.mdx index 13828a7f3e..20619a41e0 100644 --- a/docs/develop/dotnet/best-practices/converters-and-encryption.mdx +++ b/docs/develop/dotnet/best-practices/data-handling/data-encryption.mdx @@ -1,25 +1,17 @@ --- -id: converters-and-encryption -title: Converters and encryption - .NET SDK -sidebar_label: Converters and encryption -description: Use a custom Payload Codec and Converter in the .NET SDK to modify Temporal Data Conversion behavior, including examples for encryption and camel case conversion. -keywords: - - sdk - - dotnet - - data encryption - - codec server - - converter +id: data-encryption +title: Payload encryption - .NET SDK +sidebar_label: Payload encryption +toc_max_heading_level: 3 tags: - - .Net SDK - - Temporal SDKs - Security - - Codec Server - - Data Converters - Encryption + - Codec Server + - .NET SDK + - Temporal SDKs +description: Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the .NET SDK. --- -import { CaptionedImage } from '@site/src/components'; - Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. @@ -75,7 +67,7 @@ public class EncryptionCodec : IPayloadCodec } ``` -**Set Data Converter to use custom Payload Codec** +### Set Data Converter to use custom Payload Codec When creating a client, the default `DataConverter` can be updated with the payload codec like so: @@ -91,69 +83,10 @@ var myClient = await TemporalClient.ConnectAsync(new("localhost:7233") For reference, see the [Encryption](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption) sample. -### Using a Codec Server +## Using a Codec Server A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. - -## Payload conversion - -Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. - -### Conversion sequence {/* #conversion-sequence */} - -The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. -You can set multiple encoding Payload Converters to run your conversions. -When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. - -Payload Converters can be customized independently of a Payload Codec. -Temporal's Converter architecture looks like this: - - - -### Custom Payload Converter {/* #custom-payload-converter */} - -Data converters are used to convert raw Temporal payloads to/from actual .NET types. -A custom data converter can be set via the `DataConverter` option when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. -Payload converters convert .NET values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures. - -Data converters are in the `Temporalio.Converters` namespace. -The default data converter uses a default payload converter, which supports the following types: - -- `null` -- `byte[]` -- `Google.Protobuf.IMessage` instances -- Anything that `System.Text.Json` supports -- `IRawValue` as unconverted raw payloads - -Custom converters can be created for all uses. For example, to create client with a data converter that converts all C# -property names to camel case, you would: - -```csharp -using System.Text.Json; -using Temporalio.Client; -using Temporalio.Converters; - -public class CamelCasePayloadConverter : DefaultPayloadConverter -{ - public CamelCasePayloadConverter() - : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase }) - { - } -} - -var client = await TemporalClient.ConnectAsync(new() -{ - TargetHost = "localhost:7233", - Namespace = "my-namespace", - DataConverter = DataConverter.Default with { PayloadConverter = new CamelCasePayloadConverter() }, -}); -``` - - diff --git a/docs/develop/dotnet/best-practices/data-handling/index.mdx b/docs/develop/dotnet/best-practices/data-handling/index.mdx new file mode 100644 index 0000000000..12c99413d7 --- /dev/null +++ b/docs/develop/dotnet/best-practices/data-handling/index.mdx @@ -0,0 +1,36 @@ +--- +id: data-handling +title: Data handling - .NET SDK +sidebar_label: Data handling +description: + Learn how Temporal handles data through the Data Converter, including payload conversion, encryption, and large + payload storage. +toc_max_heading_level: 3 +tags: + - .NET SDK + - Temporal SDKs + - Data Converters +--- + +import { CaptionedImage } from '@site/src/components'; + +All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three +layers that handle different concerns: + + + +Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON +serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when +your application requires non-JSON types, encryption, or payload offloading. + +| | [PayloadConverter](/develop/dotnet/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/dotnet/best-practices/data-handling/data-encryption) | +| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | +| **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | +| **Default** | JSON serialization | None (passthrough) | + +For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). diff --git a/docs/develop/dotnet/best-practices/index.mdx b/docs/develop/dotnet/best-practices/index.mdx index 5cb9b95f4a..370f01f597 100644 --- a/docs/develop/dotnet/best-practices/index.mdx +++ b/docs/develop/dotnet/best-practices/index.mdx @@ -24,4 +24,4 @@ import * as Components from '@site/src/components'; - [Error handling](/develop/dotnet/best-practices/error-handling) - [Testing](/develop/dotnet/best-practices/testing-suite) - [Debugging](/develop/dotnet/best-practices/debugging) -- [Converters and encryption](/develop/dotnet/best-practices/converters-and-encryption) +- [Converters and encryption](/develop/dotnet/best-practices/data-handling) diff --git a/docs/develop/dotnet/index.mdx b/docs/develop/dotnet/index.mdx index 6f2e6a0fdf..7a137ead22 100644 --- a/docs/develop/dotnet/index.mdx +++ b/docs/develop/dotnet/index.mdx @@ -84,7 +84,7 @@ From there, you can dive deeper into any of the Temporal primitives to start bui - [Error handling](/develop/dotnet/best-practices/error-handling) - [Testing](/develop/dotnet/best-practices/testing-suite) - [Debugging](/develop/dotnet/best-practices/debugging) -- [Converters and encryption](/develop/dotnet/best-practices/converters-and-encryption) +- [Converters and encryption](/develop/dotnet/best-practices/data-handling) ## Temporal .NET technical resources diff --git a/docs/develop/java/best-practices/data-handling/data-conversion.mdx b/docs/develop/java/best-practices/data-handling/data-conversion.mdx new file mode 100644 index 0000000000..d7c4981675 --- /dev/null +++ b/docs/develop/java/best-practices/data-handling/data-conversion.mdx @@ -0,0 +1,107 @@ +--- +id: data-conversion +title: Payload conversion - Java SDK +sidebar_label: Payload conversion +toc_max_heading_level: 3 +tags: + - Data Converters + - Java SDK + - Temporal SDKs +description: Customize how Temporal serializes application objects using Payload Converters in the Java SDK. +--- + +import { CaptionedImage } from '@site/src/components'; + +Payload Converters serialize your application objects into a `Payload` and deserialize them back. +A `Payload` is a binary form with metadata that Temporal uses to transport data. + +By default, Temporal uses a Payload Converter that handles `null`, byte arrays, protobuf messages, and anything JSON-serializable. +You only need a custom Payload Converter when your application uses types that aren't natively supported. + +## Default supported types + +The default Data Converter supports converting multiple types including: + +- `null` +- Byte arrays +- Protobuf JSON: if a value is an instance of a Protobuf message, it is encoded with proto3 JSON +- [Jackson JSON](https://docs.spring.io/spring-framework/reference/web/webmvc/mvc-controller/ann-methods/jackson.html) +- Anything that can be converted to JSON + +## Using custom Payload conversion {/* #custom-payload-conversion */} + +Temporal SDKs provide a [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. + +Implementing custom Payload conversion is optional. +It is needed only if the [default Data Converter](/default-custom-data-converters#default-data-converter) does not support your custom values. + +To support custom Payload conversion, create a [custom Payload Converter](/payload-converter#composite-data-converters) and configure the Data Converter to use it in your Client options. + +The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. +You can set multiple encoding Payload Converters to run your conversions. +When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. + +Payload Converters can be customized independently of a Payload Codec. +Temporal's Converter architecture looks like this: + + + +Create a custom implementation of a [PayloadConverter](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/PayloadConverter.html) interface and use the `withPayloadConverterOverrides` method to implement the custom object conversion with `DefaultDataConverter`. + +`PayloadConverter` serializes and deserializes method parameters that need to be sent over the wire. +You can create a custom implementation of `PayloadConverter` for custom formats, as shown in the following example: + +```java +/** Payload Converter specific to your custom object */ +public class YourCustomPayloadConverter implements PayloadConverter { + //... + @Override + public String getEncodingType() { + return "json/plain"; // The encoding type determines which default conversion behavior to override. + } + + @Override + public Optional toData(Object value) throws DataConverterException { + // Add your convert-to logic here. + } + + @Override + public T fromData(Payload content, Class valueClass, Type valueType) + throws DataConverterException { + // Add your convert-from logic here. + } +//... +} +``` + +You can also use [specific implementation classes](https://www.javadoc.io/static/io.temporal/temporal-sdk/1.18.1/io/temporal/common/converter/package-summary.html) provided in the Java SDK. + +For example, to create a custom `JacksonJsonPayloadConverter`, use the following: + +```java +//... +private static JacksonJsonPayloadConverter yourCustomJacksonJsonPayloadConverter() { + ObjectMapper objectMapper = new ObjectMapper(); + // Add your custom logic here. + return new JacksonJsonPayloadConverter(objectMapper); +} +//... +``` + +To set your custom Payload Converter, use it with [withPayloadConverterOverrides](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DefaultDataConverter.html#withPayloadConverterOverrides(io.temporal.common.converter.PayloadConverter...)) with a new instance of `DefaultDataConverter` in your `WorkflowClient` options that you use in your Worker process and to start your Workflow Executions. + +The following example shows how to set a custom `YourCustomPayloadConverter` Payload Converter. + +```java +//... +DefaultDataConverter ddc = + DefaultDataConverter.newDefaultInstance() + .withPayloadConverterOverrides(new YourCustomPayloadConverter()); + + WorkflowClientOptions workflowClientOptions = + WorkflowClientOptions.newBuilder().setDataConverter(ddc).build(); +//... +``` diff --git a/docs/develop/java/best-practices/converters-and-encryption.mdx b/docs/develop/java/best-practices/data-handling/data-encryption.mdx similarity index 66% rename from docs/develop/java/best-practices/converters-and-encryption.mdx rename to docs/develop/java/best-practices/data-handling/data-encryption.mdx index a4a88a1744..a388ad08dd 100644 --- a/docs/develop/java/best-practices/converters-and-encryption.mdx +++ b/docs/develop/java/best-practices/data-handling/data-encryption.mdx @@ -1,25 +1,17 @@ --- -id: converters-and-encryption -title: Converters and encryption - Java SDK -sidebar_label: Converters and encryption -description: Create and implement a Custom Payload Codec and Payload Converter in Java using the Temporal SDK for custom data encryption, compression, and type conversion. -toc_max_heading_level: 4 -keywords: - - java sdk - - data converter - - payload conversion - - payload converter +id: data-encryption +title: Payload encryption - Java SDK +sidebar_label: Payload encryption +toc_max_heading_level: 3 tags: - Security - - Codec Server - - Data Converters - Encryption + - Codec Server - Java SDK - Temporal SDKs +description: Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Java SDK. --- -import { CaptionedImage } from '@site/src/components'; - Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. @@ -172,81 +164,3 @@ The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. For reference, see the [Codec server](https://github.com/temporalio/sdk-java/tree/master/temporal-remote-data-encoder) sample. - -## Using custom Payload conversion {/* #custom-payload-conversion */} - -Temporal SDKs provide a [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. - -Implementing custom Payload conversion is optional. -It is needed only if the [default Data Converter](/default-custom-data-converters#default-data-converter) does not support your custom values. - -To support custom Payload conversion, create a [custom Payload Converter](/payload-converter#composite-data-converters) and configure the Data Converter to use it in your Client options. - -The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. -You can set multiple encoding Payload Converters to run your conversions. -When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. - -Payload Converters can be customized independently of a Payload Codec. -Temporal's Converter architecture looks like this: - - - -Create a custom implementation of a [PayloadConverter](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/PayloadConverter.html) interface and use the `withPayloadConverterOverrides` method to implement the custom object conversion with `DefaultDataConverter`. - -`PayloadConverter` serializes and deserializes method parameters that need to be sent over the wire. -You can create a custom implementation of `PayloadConverter` for custom formats, as shown in the following example: - -```java -/** Payload Converter specific to your custom object */ -public class YourCustomPayloadConverter implements PayloadConverter { - //... - @Override - public String getEncodingType() { - return "json/plain"; // The encoding type determines which default conversion behavior to override. - } - - @Override - public Optional toData(Object value) throws DataConverterException { - // Add your convert-to logic here. - } - - @Override - public T fromData(Payload content, Class valueClass, Type valueType) - throws DataConverterException { - // Add your convert-from logic here. - } -//... -} -``` - -You can also use [specific implementation classes](https://www.javadoc.io/static/io.temporal/temporal-sdk/1.18.1/io/temporal/common/converter/package-summary.html) provided in the Java SDK. - -For example, to create a custom `JacksonJsonPayloadConverter`, use the following: - -```java -//... -private static JacksonJsonPayloadConverter yourCustomJacksonJsonPayloadConverter() { - ObjectMapper objectMapper = new ObjectMapper(); - // Add your custom logic here. - return new JacksonJsonPayloadConverter(objectMapper); -} -//... -``` - -To set your custom Payload Converter, use it with [withPayloadConverterOverrides](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DefaultDataConverter.html#withPayloadConverterOverrides(io.temporal.common.converter.PayloadConverter...)) with a new instance of `DefaultDataConverter` in your `WorkflowClient` options that you use in your Worker process and to start your Workflow Executions. - -The following example shows how to set a custom `YourCustomPayloadConverter` Payload Converter. - -```java -//... -DefaultDataConverter ddc = - DefaultDataConverter.newDefaultInstance() - .withPayloadConverterOverrides(new YourCustomPayloadConverter()); - - WorkflowClientOptions workflowClientOptions = - WorkflowClientOptions.newBuilder().setDataConverter(ddc).build(); -//... -``` diff --git a/docs/develop/java/best-practices/data-handling/index.mdx b/docs/develop/java/best-practices/data-handling/index.mdx new file mode 100644 index 0000000000..7108c8b639 --- /dev/null +++ b/docs/develop/java/best-practices/data-handling/index.mdx @@ -0,0 +1,36 @@ +--- +id: data-handling +title: Data handling - Java SDK +sidebar_label: Data handling +description: + Learn how Temporal handles data through the Data Converter, including payload conversion, encryption, and large + payload storage. +toc_max_heading_level: 3 +tags: + - Java SDK + - Temporal SDKs + - Data Converters +--- + +import { CaptionedImage } from '@site/src/components'; + +All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three +layers that handle different concerns: + + + +Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON +serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when +your application requires non-JSON types, encryption, or payload offloading. + +| | [PayloadConverter](/develop/java/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/java/best-practices/data-handling/data-encryption) | +| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | +| **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | +| **Default** | JSON serialization | None (passthrough) | + +For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). diff --git a/docs/develop/java/best-practices/index.mdx b/docs/develop/java/best-practices/index.mdx index b605e16474..f7798ea1fe 100644 --- a/docs/develop/java/best-practices/index.mdx +++ b/docs/develop/java/best-practices/index.mdx @@ -23,4 +23,4 @@ import * as Components from '@site/src/components'; - [Testing](/develop/java/best-practices/testing-suite) - [Debugging](/develop/java/best-practices/debugging) -- [Converters and encryption](/develop/java/best-practices/converters-and-encryption) +- [Converters and encryption](/develop/java/best-practices/data-handling) diff --git a/docs/develop/java/index.mdx b/docs/develop/java/index.mdx index 27897090d0..ac761e5552 100644 --- a/docs/develop/java/index.mdx +++ b/docs/develop/java/index.mdx @@ -81,7 +81,7 @@ From there, you can dive deeper into any of the Temporal primitives to start bui - [Testing](/develop/java/best-practices/testing-suite) - [Debugging](/develop/java/best-practices/debugging) -- [Converters and encryption](/develop/java/best-practices/converters-and-encryption) +- [Converters and encryption](/develop/java/best-practices/data-handling) ## [Integrations](/develop/java/integrations) diff --git a/docs/develop/ruby/best-practices/converters-and-encryption.mdx b/docs/develop/ruby/best-practices/converters-and-encryption.mdx deleted file mode 100644 index ec22575a65..0000000000 --- a/docs/develop/ruby/best-practices/converters-and-encryption.mdx +++ /dev/null @@ -1,175 +0,0 @@ ---- -id: converters-and-encryption -title: Converters and encryption - Ruby SDK -sidebar_label: Converters and encryption -description: Use a custom Payload Codec and Converter in the Ruby SDK to modify Temporal Data Conversion behavior, including examples for encryption and formatting. -keywords: - - sdk - - ruby - - data encryption - - codec - - converter -tags: - - Ruby SDK - - Temporal SDKs - - Security - - Codec Server - - Data Converters - - Encryption ---- - -import { CaptionedImage } from '@site/src/components'; - -Temporal's security model is designed around client-side encryption of Payloads. -A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. -This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. -It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. - -A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. -Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. - -The server itself never adds encryption over Payloads. -Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. -When working with sensitive data, you should always implement Payload encryption. - -## Custom Payload Codec {/* #custom-payload-codec */} - -Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps. -If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`. - -The Payload Codec needs to extend `Temporalio::Converters::PayloadCodec` and implement `encode` and `decode` methods. -These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field. -Do not mutate the existing payloads. -Here is an example of an encryption codec that just uses base64 in each direction: - -```ruby -class Base64Codec < Temporalio::Converters::PayloadCodec - def encode(payloads) - payloads.map do |p| - Temporalio::Api::Common::V1::Payload.new( - # Set our specific encoding. We may also want to add a key ID in here for use by - # the decode side - metadata: { 'encoding' => 'binary/my-payload-encoding' }, - data: Base64.strict_encode64(p.to_proto) - ) - end - end - - def decode(payloads) - payloads.map do |p| - # Ignore if it doesn't have our expected encoding - next p unless p.metadata['encoding'] == 'binary/my-payload-encoding' - - Temporalio::Api::Common::V1::Payload.decode( - Base64.strict_decode64(p.data) - ) - end - end -end -``` - -**Set Data Converter to use custom Payload Codec** - -When creating a client, the default `DataConverter` can be updated with the payload codec like so: - -```ruby -my_client = Temporalio::Client.connect( - 'localhost:7233', - 'my-namespace', - data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new) -) -``` - -- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. -- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. - Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output. - - - -### Using a Codec Server - -A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. -The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. -You create, operate, and manage access to your Codec Server in your own environment. -The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. -Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. - -## Payload conversion {/* #custom-payload-converter */} - -Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. - -### Conversion sequence {/* #conversion-sequence */} - -The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. -You can set multiple encoding Payload Converters to run your conversions. -When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. - -Payload Converters can be customized independently of a Payload Codec. -Temporal's Converter architecture looks like this: - - - -### Supported Data Types {/* #supported-data-types */} - -Data converters are used to convert raw Temporal payloads to/from actual Ruby types. -A custom data converter can be set via the `data_converter` keyword argument when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. -Payload converters convert Ruby values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures. - -Data converters are in the `Temporalio::Converters` module. -The default data converter uses a default payload converter, which supports the following types: - -- `nil` -- "bytes" (i.e. `String` with `Encoding::ASCII_8BIT` encoding) -- `Google::Protobuf::MessageExts` instances -- [JSON module](https://docs.ruby-lang.org/en/master/JSON.html) for everything else - -This means that normal Ruby objects will use `JSON.generate` when serializing and `JSON.parse` when deserializing (with `create_additions: true` set by default). -So a Ruby object will often appear as a hash when deserialized. -Also, hashes that are passed in with symbol keys end up with string keys when deserialized. -While "JSON Additions" are supported, it is not cross-SDK-language compatible since this is a Ruby-specific construct. - -The default payload converter is a collection of "encoding payload converters". -On serialize, each encoding converter will be tried in order until one accepts (default falls through to the JSON one). -The encoding converter sets an `encoding` metadata value which is used to know which converter to use on deserialize. -Custom encoding converters can be created, or even the entire payload converter can be replaced with a different implementation. - -**NOTE:** For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. -Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. -Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. -Instead, consider having models specific for workflows/activities and translate to/from existing models as needed. -See the next section on how to do this with ActiveModel objects. - -#### ActiveModel {/* #active-model */} - -By default, ActiveModel objects do not natively support the `JSON` module. -A mixin can be created to add this support for ActiveModel, for example: - -```ruby -module ActiveModelJSONSupport - extend ActiveSupport::Concern - include ActiveModel::Serializers::JSON - - included do - def as_json(*) - super.merge(::JSON.create_id => self.class.name) - end - - def to_json(*args) - as_json.to_json(*args) - end - - def self.json_create(object) - object = object.dup - object.delete(::JSON.create_id) - new(**object.symbolize_keys) - end - end -end -``` - -Now if `include ActiveModelJSONSupport` is present on any ActiveModel class, on serialization `to_json` will be used which will use `as_json` which calls the super `as_json` but also includes the fully qualified class name as the JSON -`create_id` key. -On deserialization, Ruby JSON then uses this key to know what class to call `json_create` on. diff --git a/docs/develop/ruby/best-practices/data-handling/data-conversion.mdx b/docs/develop/ruby/best-practices/data-handling/data-conversion.mdx new file mode 100644 index 0000000000..fc18262a2e --- /dev/null +++ b/docs/develop/ruby/best-practices/data-handling/data-conversion.mdx @@ -0,0 +1,93 @@ +--- +id: data-conversion +title: Payload conversion - Ruby SDK +sidebar_label: Payload conversion +toc_max_heading_level: 3 +tags: + - Data Converters + - Ruby SDK + - Temporal SDKs +description: Customize how Temporal serializes application objects using Payload Converters in the Ruby SDK. +--- + +import { CaptionedImage } from '@site/src/components'; + +## Payload conversion {/* #custom-payload-converter */} + +Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. + +### Conversion sequence {/* #conversion-sequence */} + +The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. +You can set multiple encoding Payload Converters to run your conversions. +When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. + +Payload Converters can be customized independently of a Payload Codec. +Temporal's Converter architecture looks like this: + + + +## Supported Data Types {/* #supported-data-types */} + +Data converters are used to convert raw Temporal payloads to/from actual Ruby types. +A custom data converter can be set via the `data_converter` keyword argument when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. +Payload converters convert Ruby values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures. + +Data converters are in the `Temporalio::Converters` module. +The default data converter uses a default payload converter, which supports the following types: + +- `nil` +- "bytes" (i.e. `String` with `Encoding::ASCII_8BIT` encoding) +- `Google::Protobuf::MessageExts` instances +- [JSON module](https://docs.ruby-lang.org/en/master/JSON.html) for everything else + +This means that normal Ruby objects will use `JSON.generate` when serializing and `JSON.parse` when deserializing (with `create_additions: true` set by default). +So a Ruby object will often appear as a hash when deserialized. +Also, hashes that are passed in with symbol keys end up with string keys when deserialized. +While "JSON Additions" are supported, it is not cross-SDK-language compatible since this is a Ruby-specific construct. + +The default payload converter is a collection of "encoding payload converters". +On serialize, each encoding converter will be tried in order until one accepts (default falls through to the JSON one). +The encoding converter sets an `encoding` metadata value which is used to know which converter to use on deserialize. +Custom encoding converters can be created, or even the entire payload converter can be replaced with a different implementation. + +**NOTE:** For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. +Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. +Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. +Instead, consider having models specific for workflows/activities and translate to/from existing models as needed. +See the next section on how to do this with ActiveModel objects. + +#### ActiveModel {/* #active-model */} + +By default, ActiveModel objects do not natively support the `JSON` module. +A mixin can be created to add this support for ActiveModel, for example: + +```ruby +module ActiveModelJSONSupport + extend ActiveSupport::Concern + include ActiveModel::Serializers::JSON + + included do + def as_json(*) + super.merge(::JSON.create_id => self.class.name) + end + + def to_json(*args) + as_json.to_json(*args) + end + + def self.json_create(object) + object = object.dup + object.delete(::JSON.create_id) + new(**object.symbolize_keys) + end + end +end +``` + +Now if `include ActiveModelJSONSupport` is present on any ActiveModel class, on serialization `to_json` will be used which will use `as_json` which calls the super `as_json` but also includes the fully qualified class name as the JSON +`create_id` key. +On deserialization, Ruby JSON then uses this key to know what class to call `json_create` on. diff --git a/docs/develop/ruby/best-practices/data-handling/data-encryption.mdx b/docs/develop/ruby/best-practices/data-handling/data-encryption.mdx new file mode 100644 index 0000000000..db7d551f3b --- /dev/null +++ b/docs/develop/ruby/best-practices/data-handling/data-encryption.mdx @@ -0,0 +1,87 @@ +--- +id: data-encryption +title: Payload encryption - Ruby SDK +sidebar_label: Payload encryption +toc_max_heading_level: 3 +tags: + - Security + - Encryption + - Codec Server + - Ruby SDK + - Temporal SDKs +description: Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Ruby SDK. +--- + +Temporal's security model is designed around client-side encryption of Payloads. +A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. +This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. +It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. + +A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. +Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. + +The server itself never adds encryption over Payloads. +Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. +When working with sensitive data, you should always implement Payload encryption. + +## Custom Payload Codec {/* #custom-payload-codec */} + +Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps. +If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`. + +The Payload Codec needs to extend `Temporalio::Converters::PayloadCodec` and implement `encode` and `decode` methods. +These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field. +Do not mutate the existing payloads. +Here is an example of an encryption codec that just uses base64 in each direction: + +```ruby +class Base64Codec < Temporalio::Converters::PayloadCodec + def encode(payloads) + payloads.map do |p| + Temporalio::Api::Common::V1::Payload.new( + # Set our specific encoding. We may also want to add a key ID in here for use by + # the decode side + metadata: { 'encoding' => 'binary/my-payload-encoding' }, + data: Base64.strict_encode64(p.to_proto) + ) + end + end + + def decode(payloads) + payloads.map do |p| + # Ignore if it doesn't have our expected encoding + next p unless p.metadata['encoding'] == 'binary/my-payload-encoding' + + Temporalio::Api::Common::V1::Payload.decode( + Base64.strict_decode64(p.data) + ) + end + end +end +``` + +### Set Data Converter to use custom Payload Codec + +When creating a client, the default `DataConverter` can be updated with the payload codec like so: + +```ruby +my_client = Temporalio::Client.connect( + 'localhost:7233', + 'my-namespace', + data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new) +) +``` + +- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. +- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. + Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output. + +For reference, see the [Encryption](https://github.com/temporalio/samples-ruby/tree/main/encryption) sample. + +## Using a Codec Server + +A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. +The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. +You create, operate, and manage access to your Codec Server in your own environment. +The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. +Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. diff --git a/docs/develop/ruby/best-practices/data-handling/index.mdx b/docs/develop/ruby/best-practices/data-handling/index.mdx new file mode 100644 index 0000000000..c2728f52bd --- /dev/null +++ b/docs/develop/ruby/best-practices/data-handling/index.mdx @@ -0,0 +1,36 @@ +--- +id: data-handling +title: Data handling - Ruby SDK +sidebar_label: Data handling +description: + Learn how Temporal handles data through the Data Converter, including payload conversion, encryption, and large + payload storage. +toc_max_heading_level: 3 +tags: + - Ruby SDK + - Temporal SDKs + - Data Converters +--- + +import { CaptionedImage } from '@site/src/components'; + +All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three +layers that handle different concerns: + + + +Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON +serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when +your application requires non-JSON types, encryption, or payload offloading. + +| | [PayloadConverter](/develop/ruby/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/ruby/best-practices/data-handling/data-encryption) | +| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | +| **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | +| **Default** | JSON serialization | None (passthrough) | + +For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). diff --git a/docs/develop/ruby/best-practices/index.mdx b/docs/develop/ruby/best-practices/index.mdx index 1dfa5124c2..5c0e2656f9 100644 --- a/docs/develop/ruby/best-practices/index.mdx +++ b/docs/develop/ruby/best-practices/index.mdx @@ -24,4 +24,4 @@ import * as Components from '@site/src/components'; - [Error handling](/develop/ruby/best-practices/error-handling) - [Testing](/develop/ruby/best-practices/testing-suite) - [Debugging](/develop/ruby/best-practices/debugging) -- [Converters and encryption](/develop/ruby/best-practices/converters-and-encryption) +- [Converters and encryption](/develop/ruby/best-practices/data-handling) diff --git a/docs/develop/ruby/index.mdx b/docs/develop/ruby/index.mdx index 8588a66832..8236b0fea4 100644 --- a/docs/develop/ruby/index.mdx +++ b/docs/develop/ruby/index.mdx @@ -86,7 +86,7 @@ From there, you can dive deeper into any of the Temporal primitives to start bui - [Error handling](/develop/ruby/best-practices/error-handling) - [Testing](/develop/ruby/best-practices/testing-suite) - [Debugging](/develop/ruby/best-practices/debugging) -- [Converters and encryption](/develop/ruby/best-practices/converters-and-encryption) +- [Converters and encryption](/develop/ruby/best-practices/data-handling) ## Temporal Ruby technical resources diff --git a/docs/develop/ruby/integrations/rails-integration.mdx b/docs/develop/ruby/integrations/rails-integration.mdx index 28de875214..47711755b2 100644 --- a/docs/develop/ruby/integrations/rails-integration.mdx +++ b/docs/develop/ruby/integrations/rails-integration.mdx @@ -28,7 +28,7 @@ For ActiveRecord, or other general/ORM models that are used for a different purp Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. Instead, consider having models specific for Workflows/Activities and translate to/from existing models as needed. -See the [ActiveModel section](/develop/ruby/best-practices/converters-and-encryption#active-model) on how to do this with ActiveModel objects. +See the [ActiveModel section](/develop/ruby/best-practices/data-handling/data-conversion#active-model) on how to do this with ActiveModel objects. ## Lazy/Eager Loading diff --git a/docs/develop/typescript/best-practices/converters-and-encryption.mdx b/docs/develop/typescript/best-practices/data-handling/data-conversion.mdx similarity index 67% rename from docs/develop/typescript/best-practices/converters-and-encryption.mdx rename to docs/develop/typescript/best-practices/data-handling/data-conversion.mdx index 1bd3a149bc..9fa095df81 100644 --- a/docs/develop/typescript/best-practices/converters-and-encryption.mdx +++ b/docs/develop/typescript/best-practices/data-handling/data-conversion.mdx @@ -1,40 +1,31 @@ --- -id: converters-and-encryption -title: Converters and encryption - TypeScript SDK -sidebar_label: Converters and encryption -slug: /develop/typescript/converters-and-encryption -toc_max_heading_level: 2 -keywords: - - data-encryption +id: data-conversion +title: Payload conversion - TypeScript SDK +sidebar_label: Payload conversion +toc_max_heading_level: 3 tags: - - Security - - Codec Server - Data Converters - - Encryption - TypeScript SDK - Temporal SDKs -description: Create a custom Payload Converter in TypeScript with Temporal SDKs to handle non-JSON-serializable values, configure your Data Converter, and use protobufs and encryption seamlessly in your Workflows and Activities. +description: Customize how Temporal serializes application objects using Payload Converters in the TypeScript SDK. --- -## Payload Converter and Payload Codec Summary +Payload Converters serialize your application objects into a `Payload` and deserialize them back. +A `Payload` is a binary form with metadata that Temporal uses to transport data. -This section summarizes the difference between a Payload Converter and Payload Codec. +By default, Temporal uses a Payload Converter that handles `null`, byte arrays, protobuf messages, and anything JSON-serializable. +You only need a custom Payload Converter when your application uses types that aren't natively supported. -### Payload Converter +## Default supported types -Payload Converters are responsible for serializing application objects into a Payload and deserializing them back into application objects. A Payload, in this context, is a binary form suitable for network transmission that may include some metadata. This serialization process transforms an object (like those in JSON or Protobuf formats) into a binary format and vice versa. For example, an object might be serialized to JSON with UTF-8 byte encoding or to a protobuf binary using a specific set of protobuf message definitions. +The default Data Converter supports converting multiple types including: -Due to their operation within the Workflow context, Payload Converters run inside the Workflow sandbox. Consequently, Payload Converters cannot access external services or employ non-deterministic modules, which excludes most types of encryption due to their non-deterministic nature. +- `undefined` +- `null` +- `Uint8Array` +- JSON serializables -### Payload Codec - -Payload Codecs transform one Payload into another, converting binary data to a different binary format. Unlike Payload Converters, Payload Codecs do not operate within the Workflow sandbox. This allows them to execute operations that can include calls to remote services and the use of non-deterministic modules, which are critical for tasks such as encrypting Payloads, compressing data, or offloading large payloads to an object store. Payload Codecs can also be implemented as a Codec Server (which will be described later on). - -### Operational Chain - -In practice, these two components operate in a chain to handle data securely. Incoming data first passes through a Payload Converter through the `toPayload` method, turning application objects into Payloads. These Payloads are then processed by the Payload Codec through the `encode` method, which adjusts the Payload according to the required security or efficiency needs before it is sent to the Temporal Cluster. - -The process is symmetric for outgoing data. Payloads retrieved from the Temporal Cluster first pass through the Payload Codec through the `decode` method, which reverses any transformations applied during encoding. Finally, the resulting Payload is converted back into an application object by the Payload Converter through the `fromPayload` method, making it ready for use within the application. +You can work with [protobufs](#protobufs) like `proto3-json-serializer` or `protobufjs` by using an instance of `DefaultPayloadConverterWithProtobufs`. ## Payload Codec @@ -460,144 +451,3 @@ export async function protoActivity(input: foo.bar.ProtoInput): Promise -#### Encryption - -> Background: [Encryption](/payload-codec#encryption) - -The following is an example class that implements the `PayloadCodec` interface: - - -[encryption/src/encryption-codec.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/encryption-codec.ts) -```ts -import { webcrypto as crypto } from 'node:crypto'; -import { METADATA_ENCODING_KEY, Payload, PayloadCodec, ValueError } from '@temporalio/common'; -import { temporal } from '@temporalio/proto'; -import { decode, encode } from '@temporalio/common/lib/encoding'; -import { decrypt, encrypt } from './crypto'; - -const ENCODING = 'binary/encrypted'; -const METADATA_ENCRYPTION_KEY_ID = 'encryption-key-id'; - -export class EncryptionCodec implements PayloadCodec { - constructor( - protected readonly keys: Map, - protected readonly defaultKeyId: string, - ) {} - - static async create(keyId: string): Promise { - const keys = new Map(); - keys.set(keyId, await fetchKey(keyId)); - return new this(keys, keyId); - } - - async encode(payloads: Payload[]): Promise { - return Promise.all( - payloads.map(async (payload) => ({ - metadata: { - [METADATA_ENCODING_KEY]: encode(ENCODING), - [METADATA_ENCRYPTION_KEY_ID]: encode(this.defaultKeyId), - }, - // Encrypt entire payload, preserving metadata - data: await encrypt( - temporal.api.common.v1.Payload.encode(payload).finish(), - this.keys.get(this.defaultKeyId)!, // eslint-disable-line @typescript-eslint/no-non-null-assertion - ), - })), - ); - } - - async decode(payloads: Payload[]): Promise { - return Promise.all( - payloads.map(async (payload) => { - if (!payload.metadata || decode(payload.metadata[METADATA_ENCODING_KEY]) !== ENCODING) { - return payload; - } - if (!payload.data) { - throw new ValueError('Payload data is missing'); - } - - const keyIdBytes = payload.metadata[METADATA_ENCRYPTION_KEY_ID]; - if (!keyIdBytes) { - throw new ValueError('Unable to decrypt Payload without encryption key id'); - } - - const keyId = decode(keyIdBytes); - let key = this.keys.get(keyId); - if (!key) { - key = await fetchKey(keyId); - this.keys.set(keyId, key); - } - const decryptedPayloadBytes = await decrypt(payload.data, key); - console.log('Decrypting payload.data:', payload.data); - return temporal.api.common.v1.Payload.decode(decryptedPayloadBytes); - }), - ); - } -} - -async function fetchKey(_keyId: string): Promise { - // In production, fetch key from a key management system (KMS). You may want to memoize requests if you'll be decoding - // Payloads that were encrypted using keys other than defaultKeyId. - const key = Buffer.from('test-key-test-key-test-key-test!'); - const cryptoKey = await crypto.subtle.importKey( - 'raw', - key, - { - name: 'AES-GCM', - }, - true, - ['encrypt', 'decrypt'], - ); - - return cryptoKey; -} -``` - - -The encryption and decryption code is in [src/crypto.ts](https://github.com/temporalio/samples-typescript/tree/main/encryption/src/crypto.ts). -Because encryption is CPU intensive, and doing AES with the crypto module built into Node.js blocks the main thread, we use `@ronomon/crypto-async`, which uses the Node.js thread pool. - -As before, we provide a custom Data Converter to the Client and Worker: - - -[encryption/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/client.ts) -```ts -const client = new Client({ - connection, - dataConverter: await getDataConverter(), -}); - -const handle = await client.workflow.start(example, { - args: ['Alice: Private message for Bob.'], - taskQueue: 'encryption', - workflowId: `my-business-id-${uuid()}`, -}); - -console.log(`Started workflow ${handle.workflowId}`); -console.log(await handle.result()); -``` - - - -[encryption/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/worker.ts) -```ts -const worker = await Worker.create({ - workflowsPath: require.resolve('./workflows'), - taskQueue: 'encryption', - dataConverter: await getDataConverter(), -}); -``` - - -When the Client sends `'Alice: Private message for Bob.'` to the Workflow, it gets encrypted on the Client and decrypted in the Worker. -The Workflow receives the decrypted message and appends another message. -When it returns that longer string, the string gets encrypted by the Worker and decrypted by the Client. - - -[encryption/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/workflows.ts) -```ts -export async function example(message: string): Promise { - return `${message}\nBob: Hi Alice, I'm Workflow Bob.`; -} -``` - diff --git a/docs/develop/typescript/best-practices/data-handling/data-encryption.mdx b/docs/develop/typescript/best-practices/data-handling/data-encryption.mdx new file mode 100644 index 0000000000..71ded79814 --- /dev/null +++ b/docs/develop/typescript/best-practices/data-handling/data-encryption.mdx @@ -0,0 +1,161 @@ +--- +id: data-encryption +title: Payload encryption - TypeScript SDK +sidebar_label: Payload encryption +toc_max_heading_level: 3 +tags: + - Security + - Encryption + - Codec Server + - TypeScript SDK + - Temporal SDKs +description: Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the TypeScript SDK. +--- + +Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. + +A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. + +The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. + +## Custom Payload Codec + +> Background: [Encryption](/payload-codec#encryption) + +The following is an example class that implements the `PayloadCodec` interface: + + +[encryption/src/encryption-codec.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/encryption-codec.ts) +```ts +import { webcrypto as crypto } from 'node:crypto'; +import { METADATA_ENCODING_KEY, Payload, PayloadCodec, ValueError } from '@temporalio/common'; +import { temporal } from '@temporalio/proto'; +import { decode, encode } from '@temporalio/common/lib/encoding'; +import { decrypt, encrypt } from './crypto'; + +const ENCODING = 'binary/encrypted'; +const METADATA_ENCRYPTION_KEY_ID = 'encryption-key-id'; + +export class EncryptionCodec implements PayloadCodec { + constructor( + protected readonly keys: Map, + protected readonly defaultKeyId: string, + ) {} + + static async create(keyId: string): Promise { + const keys = new Map(); + keys.set(keyId, await fetchKey(keyId)); + return new this(keys, keyId); + } + + async encode(payloads: Payload[]): Promise { + return Promise.all( + payloads.map(async (payload) => ({ + metadata: { + [METADATA_ENCODING_KEY]: encode(ENCODING), + [METADATA_ENCRYPTION_KEY_ID]: encode(this.defaultKeyId), + }, + // Encrypt entire payload, preserving metadata + data: await encrypt( + temporal.api.common.v1.Payload.encode(payload).finish(), + this.keys.get(this.defaultKeyId)!, // eslint-disable-line @typescript-eslint/no-non-null-assertion + ), + })), + ); + } + + async decode(payloads: Payload[]): Promise { + return Promise.all( + payloads.map(async (payload) => { + if (!payload.metadata || decode(payload.metadata[METADATA_ENCODING_KEY]) !== ENCODING) { + return payload; + } + if (!payload.data) { + throw new ValueError('Payload data is missing'); + } + + const keyIdBytes = payload.metadata[METADATA_ENCRYPTION_KEY_ID]; + if (!keyIdBytes) { + throw new ValueError('Unable to decrypt Payload without encryption key id'); + } + + const keyId = decode(keyIdBytes); + let key = this.keys.get(keyId); + if (!key) { + key = await fetchKey(keyId); + this.keys.set(keyId, key); + } + const decryptedPayloadBytes = await decrypt(payload.data, key); + console.log('Decrypting payload.data:', payload.data); + return temporal.api.common.v1.Payload.decode(decryptedPayloadBytes); + }), + ); + } +} + +async function fetchKey(_keyId: string): Promise { + // In production, fetch key from a key management system (KMS). You may want to memoize requests if you'll be decoding + // Payloads that were encrypted using keys other than defaultKeyId. + const key = Buffer.from('test-key-test-key-test-key-test!'); + const cryptoKey = await crypto.subtle.importKey( + 'raw', + key, + { + name: 'AES-GCM', + }, + true, + ['encrypt', 'decrypt'], + ); + + return cryptoKey; +} +``` + + +The encryption and decryption code is in [src/crypto.ts](https://github.com/temporalio/samples-typescript/tree/main/encryption/src/crypto.ts). +Because encryption is CPU intensive, and doing AES with the crypto module built into Node.js blocks the main thread, we use `@ronomon/crypto-async`, which uses the Node.js thread pool. + +As before, we provide a custom Data Converter to the Client and Worker: + + +[encryption/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/client.ts) +```ts +const client = new Client({ + connection, + dataConverter: await getDataConverter(), +}); + +const handle = await client.workflow.start(example, { + args: ['Alice: Private message for Bob.'], + taskQueue: 'encryption', + workflowId: `my-business-id-${uuid()}`, +}); + +console.log(`Started workflow ${handle.workflowId}`); +console.log(await handle.result()); +``` + + + +[encryption/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/worker.ts) +```ts +const worker = await Worker.create({ + workflowsPath: require.resolve('./workflows'), + taskQueue: 'encryption', + dataConverter: await getDataConverter(), +}); +``` + + +When the Client sends `'Alice: Private message for Bob.'` to the Workflow, it gets encrypted on the Client and decrypted in the Worker. +The Workflow receives the decrypted message and appends another message. +When it returns that longer string, the string gets encrypted by the Worker and decrypted by the Client. + + +[encryption/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/workflows.ts) +```ts +export async function example(message: string): Promise { + return `${message}\nBob: Hi Alice, I'm Workflow Bob.`; +} +``` + diff --git a/docs/develop/typescript/best-practices/data-handling/index.mdx b/docs/develop/typescript/best-practices/data-handling/index.mdx new file mode 100644 index 0000000000..fab34802e6 --- /dev/null +++ b/docs/develop/typescript/best-practices/data-handling/index.mdx @@ -0,0 +1,36 @@ +--- +id: data-handling +title: Data handling - TypeScript SDK +sidebar_label: Data handling +description: + Learn how Temporal handles data through the Data Converter, including payload conversion, encryption, and large + payload storage. +toc_max_heading_level: 3 +tags: + - TypeScript SDK + - Temporal SDKs + - Data Converters +--- + +import { CaptionedImage } from '@site/src/components'; + +All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three +layers that handle different concerns: + + + +Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON +serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when +your application requires non-JSON types, encryption, or payload offloading. + +| | [PayloadConverter](/develop/typescript/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/typescript/best-practices/data-handling/data-encryption) | +| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | +| **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | +| **Default** | JSON serialization | None (passthrough) | + +For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). diff --git a/docs/develop/typescript/best-practices/index.mdx b/docs/develop/typescript/best-practices/index.mdx index e2b83d741a..ea81c5a102 100644 --- a/docs/develop/typescript/best-practices/index.mdx +++ b/docs/develop/typescript/best-practices/index.mdx @@ -24,5 +24,5 @@ import * as Components from '@site/src/components'; - [Testing](/develop/typescript/best-practices/testing-suite) - [Debugging](/develop/typescript/best-practices/debugging) -- [Converters and encryption](/develop/typescript/converters-and-encryption) +- [Converters and encryption](/develop/typescript/best-practices/data-handling) - [Entity pattern](/develop/typescript/best-practices/entity-pattern) diff --git a/docs/develop/typescript/index.mdx b/docs/develop/typescript/index.mdx index f14efd7b02..c2af1b4f8b 100644 --- a/docs/develop/typescript/index.mdx +++ b/docs/develop/typescript/index.mdx @@ -81,7 +81,7 @@ Once your local Temporal Service is set up, continue building with the following - [Testing](/develop/typescript/best-practices/testing-suite) - [Debugging](/develop/typescript/best-practices/debugging) -- [Converters and encryption](/develop/typescript/converters-and-encryption) +- [Converters and encryption](/develop/typescript/best-practices/data-handling) - [Entity pattern](/develop/typescript/best-practices/entity-pattern) ## [Integrations](/develop/typescript/integrations) diff --git a/docs/encyclopedia/data-conversion/payload-codec.mdx b/docs/encyclopedia/data-conversion/payload-codec.mdx index 9d9636c24f..c7cf4f45b3 100644 --- a/docs/encyclopedia/data-conversion/payload-codec.mdx +++ b/docs/encyclopedia/data-conversion/payload-codec.mdx @@ -24,13 +24,20 @@ This page discusses [Payload Codec](#payload-codec). ## What is a Payload Codec? {/* #payload-codec */} -A Payload Codec transforms an array of [Payloads](/dataconversion#payload) (for example, a list of Workflow arguments) into another array of Payloads. +A Payload Codec transforms an array of [Payloads](/dataconversion#payload) (for example, a list of Workflow arguments) into another array of Payloads. Unlike Payload Converters, Payload Codecs do not operate within the Workflow sandbox. -When serializing to Payloads, the Payload Converter is applied first to convert your objects to bytes, followed by codecs that convert bytes to bytes. -When deserializing from Payloads, codecs are applied first to last to reverse the effect, followed by the Payload Converter. +This allows them to execute operations that can include calls to remote services and the use of non-deterministic modules, which are critical for tasks such as encrypting Payloads, compressing data, or offloading large payloads to an object store. Payload Codecs can also be implemented as a Codec Server. + +When serializing to Payloads, the Payload Converter is applied first to convert your objects to bytes, followed by codecs that convert bytes to bytes. When deserializing from Payloads, codecs are applied first to last to reverse the effect, followed by the Payload Converter. Use a custom Payload Codec to transform your Payloads; for example, implementing compression and/or encryption on your Workflow Execution data. +### Operational Chain + +In practice, the [Payload Converter](/payload-converter) and Payload Codex operate in a chain to handle data securely. Incoming data first passes through a Payload Converter through the `toPayload` method, turning application objects into Payloads. These Payloads are then processed by the Payload Codec through the `encode` method, which adjusts the Payload according to the required security or efficiency needs before it is sent to the Temporal Cluster. + +The process is symmetric for outgoing data. Payloads retrieved from the Temporal Cluster first pass through the Payload Codec through the `decode` method, which reverses any transformations applied during encoding. Finally, the resulting Payload is converted back into an application object by the Payload Converter through the `fromPayload` method, making it ready for use within the application. + ### Encryption {/* #encryption */} Using end-to-end encryption in your custom Data Converter ensures that sensitive application data is secure when handled by the Temporal Server. diff --git a/docs/encyclopedia/data-conversion/payload-converter.mdx b/docs/encyclopedia/data-conversion/payload-converter.mdx index 099a3de4b9..aca4a40c98 100644 --- a/docs/encyclopedia/data-conversion/payload-converter.mdx +++ b/docs/encyclopedia/data-conversion/payload-converter.mdx @@ -24,10 +24,13 @@ This page discusses [Payload Converter](#payload-converter). ## What is a Payload Converter? {/* #payload-converter */} -A Payload Converter serializes data, converting values to bytes and back. +A Payload Converter serializes data, converting values to bytes and back. This is how you can serialize application objects into a Payload and deserialize them back into application objects. -When you initiate a Workflow Execution through a Client and pass data as input, the input is serialized using a Data Converter that runs it through a set of Payload Converters. -When your Workflow Execution starts, this data input is deserialized and passed as input to your Workflow. +A Payload, in this context, is a binary form suitable for network transmission that may include some metadata. This serialization process transforms an object (like those in JSON or Protobuf formats) into a binary format and vice versa. For example, an object might be serialized to JSON with UTF-8 byte encoding or to a protobuf binary using a specific set of protobuf message definitions. + +When you initiate a Workflow Execution through a Client and pass data as input, the input is serialized using a Data Converter that runs it through a set of Payload Converters. When your Workflow Execution starts, this data input is deserialized and passed as input to your Workflow. + +Due to their operation within the Workflow context, Payload Converters run inside the Workflow sandbox. Consequently, Payload Converters cannot access external services or employ non-deterministic modules, which excludes most types of encryption due to their non-deterministic nature. ### Composite Data Converters {/* #composite-data-converters */} diff --git a/docs/evaluate/development-production-features/data-encryption.mdx b/docs/evaluate/development-production-features/data-encryption.mdx index 57dd6e75cf..5e91f4677f 100644 --- a/docs/evaluate/development-production-features/data-encryption.mdx +++ b/docs/evaluate/development-production-features/data-encryption.mdx @@ -30,9 +30,9 @@ Jump straight to a Temporal SDK feature guide. - + - - - + + + diff --git a/readme/ZOOMABLE_IMAGES.md b/readme/ZOOMABLE_IMAGES.md index 586ad5b28e..00b7e6dba1 100644 --- a/readme/ZOOMABLE_IMAGES.md +++ b/readme/ZOOMABLE_IMAGES.md @@ -137,7 +137,7 @@ Pages that did **not** previously have any zoom but contain raster images ≥ 80 - [/develop/dotnet/client/temporal-client](https://docs.temporal.io/develop/dotnet/client/temporal-client) — _max 4096px_ - [/develop/dotnet/nexus/feature-guide](https://docs.temporal.io/develop/dotnet/nexus/feature-guide) — _max 3542px_ -- [/develop/dotnet/best-practices/converters-and-encryption](https://docs.temporal.io/develop/dotnet/best-practices/converters-and-encryption) — _max 1320px_ +- [/develop/dotnet/best-practices/data-handling](https://docs.temporal.io/develop/dotnet/best-practices/data-handling) — _max 1320px_ #### Develop — go @@ -149,7 +149,7 @@ Pages that did **not** previously have any zoom but contain raster images ≥ 80 - [/develop/java/client/temporal-client](https://docs.temporal.io/develop/java/client/temporal-client) — _max 4096px_ - [/develop/java/nexus/feature-guide](https://docs.temporal.io/develop/java/nexus/feature-guide) — _max 3542px_ -- [/develop/java/best-practices/converters-and-encryption](https://docs.temporal.io/develop/java/best-practices/converters-and-encryption) — _max 1320px_ +- [/develop/java/best-practices/data-handling](https://docs.temporal.io/develop/java/best-practices/data-handling) — _max 1320px_ #### Develop — python @@ -159,7 +159,7 @@ Pages that did **not** previously have any zoom but contain raster images ≥ 80 #### Develop — ruby - [/develop/ruby/client/temporal-client](https://docs.temporal.io/develop/ruby/client/temporal-client) — _max 4096px_ -- [/develop/ruby/best-practices/converters-and-encryption](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption) — _max 1320px_ +- [/develop/ruby/best-practices/data-handling](https://docs.temporal.io/develop/ruby/best-practices/data-handling) — _max 1320px_ #### Develop — typescript diff --git a/sidebars.js b/sidebars.js index aea9c0cb3b..8e8684a1bd 100644 --- a/sidebars.js +++ b/sidebars.js @@ -343,7 +343,19 @@ module.exports = { items: [ 'develop/java/best-practices/testing-suite', 'develop/java/best-practices/debugging', - 'develop/java/best-practices/converters-and-encryption', + { + type: 'category', + label: 'Data handling', + collapsed: true, + link: { + type: 'doc', + id: 'develop/java/best-practices/data-handling/data-handling', + }, + items: [ + 'develop/java/best-practices/data-handling/data-conversion', + 'develop/java/best-practices/data-handling/data-encryption', + ], + }, ], }, { @@ -726,8 +738,20 @@ module.exports = { items: [ 'develop/typescript/best-practices/testing-suite', 'develop/typescript/best-practices/debugging', - 'develop/typescript/best-practices/converters-and-encryption', 'develop/typescript/best-practices/entity-pattern', + { + type: 'category', + label: 'Data handling', + collapsed: true, + link: { + type: 'doc', + id: 'develop/typescript/best-practices/data-handling/data-handling', + }, + items: [ + 'develop/typescript/best-practices/data-handling/data-conversion', + 'develop/typescript/best-practices/data-handling/data-encryption', + ], + }, ], }, { @@ -860,7 +884,19 @@ module.exports = { 'develop/dotnet/best-practices/error-handling', 'develop/dotnet/best-practices/testing-suite', 'develop/dotnet/best-practices/debugging', - 'develop/dotnet/best-practices/converters-and-encryption', + { + type: 'category', + label: 'Data handling', + collapsed: true, + link: { + type: 'doc', + id: 'develop/dotnet/best-practices/data-handling/data-handling', + }, + items: [ + 'develop/dotnet/best-practices/data-handling/data-conversion', + 'develop/dotnet/best-practices/data-handling/data-encryption', + ], + }, ], }, ], @@ -977,7 +1013,19 @@ module.exports = { 'develop/ruby/best-practices/error-handling', 'develop/ruby/best-practices/testing-suite', 'develop/ruby/best-practices/debugging', - 'develop/ruby/best-practices/converters-and-encryption', + { + type: 'category', + label: 'Data handling', + collapsed: true, + link: { + type: 'doc', + id: 'develop/ruby/best-practices/data-handling/data-handling', + }, + items: [ + 'develop/ruby/best-practices/data-handling/data-conversion', + 'develop/ruby/best-practices/data-handling/data-encryption', + ], + }, ], }, ], diff --git a/vercel.json b/vercel.json index 4fdf25077d..8fb39adb4a 100644 --- a/vercel.json +++ b/vercel.json @@ -2182,6 +2182,26 @@ "source": "/develop/ruby/workflows/versioning#ruby-sdk-patching-api", "destination": "/develop/ruby/workflows/versioning#patching", "permanent": true + }, + { + "source": "/develop/dotnet/best-practices/converters-and-encryption", + "destination": "/develop/dotnet/best-practices/data-handling", + "permanent": true + }, + { + "source": "/develop/java/best-practices/converters-and-encryption", + "destination": "/develop/java/best-practices/data-handling", + "permanent": true + }, + { + "source": "/develop/ruby/best-practices/converters-and-encryption", + "destination": "/develop/ruby/best-practices/data-handling", + "permanent": true + }, + { + "source": "/develop/typescript/converters-and-encryption", + "destination": "/develop/typescript/best-practices/data-handling", + "permanent": true } ] }