<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://christoph-sens.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://christoph-sens.github.io/" rel="alternate" type="text/html" /><updated>2026-09-26T16:11:58+00:00</updated><id>https://christoph-sens.github.io/feed.xml</id><title type="html">Christoph Sens</title><subtitle>Notes on Kotlin, Java and AWS from a freelance developer and consultant.</subtitle><author><name>Christoph Sens</name></author><entry><title type="html">Large SQS and SNS messages in Kotlin: the extended client pattern without Java baggage</title><link href="https://christoph-sens.github.io/2026/09/large-sqs-sns-messages-in-kotlin/" rel="alternate" type="text/html" title="Large SQS and SNS messages in Kotlin: the extended client pattern without Java baggage" /><published>2026-09-26T00:00:00+00:00</published><updated>2026-09-26T00:00:00+00:00</updated><id>https://christoph-sens.github.io/2026/09/large-sqs-sns-messages-in-kotlin</id><content type="html" xml:base="https://christoph-sens.github.io/2026/09/large-sqs-sns-messages-in-kotlin/"><![CDATA[<p>Every SQS queue and SNS topic has a message size limit. When a payload is larger, the usual answer
is the <em>claim check</em> pattern: put the payload in S3, send a small pointer instead, and resolve the
pointer on the receiving side. AWS ships this pattern as two Java libraries,
<a href="https://github.com/awslabs/amazon-sqs-java-extended-client-lib">amazon-sqs-java-extended-client-lib</a>
and <a href="https://github.com/awslabs/amazon-sns-java-extended-client-lib">amazon-sns-java-extended-client-lib</a>,
both built on <a href="https://github.com/awslabs/payload-offloading-java-common-lib-for-aws">payload-offloading-java-common-lib-for-aws</a>.</p>

<p>If your service is written in Kotlin on <a href="https://github.com/awslabs/aws-sdk-kotlin">aws-sdk-kotlin</a>,
those libraries don’t fit: they wrap the AWS SDK for Java v2 clients, not the Kotlin ones. So I
ported them. The result is three small libraries on Maven Central:</p>

<ul>
  <li><a href="https://github.com/christoph-sens/s3overflow">s3overflow</a>: the payload store (S3 upload, pointer, download, delete)</li>
  <li><a href="https://github.com/christoph-sens/sqsoverflow">sqsoverflow</a>: an <code class="language-plaintext highlighter-rouge">SqsClient</code> that offloads large message bodies</li>
  <li><a href="https://github.com/christoph-sens/snsoverflow">snsoverflow</a>: an <code class="language-plaintext highlighter-rouge">SnsClient</code> that offloads large message bodies</li>
</ul>

<p>This post covers why the Java libraries are a liability today, how the ports work, and what you
need to know before migrating.</p>

<h2 id="where-the-java-libraries-stand-today">Where the Java libraries stand today</h2>

<p>The extended clients do their job, but their foundation has aged:</p>

<ul>
  <li><strong>Old dependencies with known advisories.</strong> Both clients depend on
<code class="language-plaintext highlighter-rouge">payloadoffloading-common</code> 2.2.0, whose last release was in March 2024. It pins
<code class="language-plaintext highlighter-rouge">jackson-databind</code> 2.15.2 and <code class="language-plaintext highlighter-rouge">jackson-core</code> 2.16.0. As of September 2026, the GitHub Advisory
Database lists seven advisories affecting exactly these versions, three of them rated high,
among them bypasses of Jackson’s polymorphic type validation and a denial-of-service issue in the
async parser. Even the latest SQS client release (2.1.3, July 2026) still pulls in these versions
through the common library.</li>
  <li><strong>Java 8 bytecode and a pre-coroutine design.</strong> All three libraries compile for Java 8 and keep
separate sync and async client classes with largely duplicated pass-through code.</li>
  <li><strong>Slow release cadence.</strong> The SNS client’s latest release (2.1.0) is from March 2024, the common
library’s from the same month.</li>
</ul>

<p>To be fair: whether an advisory is exploitable depends on how Jackson is used, and the extended
clients only (de)serialize a small pointer object. You can also override the Jackson versions in
your own build. But every dependency scanner flags these versions, someone has to triage the
findings again and again, and forcing newer Jackson versions under a library that was never tested
against them is its own risk. Removing Jackson from this code path makes the question disappear.</p>

<h2 id="what-the-size-limits-are-today">What the size limits are today</h2>

<p>The limits moved recently, which changes when offloading is needed at all:</p>

<ul>
  <li><strong>SQS</strong> accepts messages up to <strong>1 MiB</strong> (body plus attributes). Until August 2025 the limit was 256 KiB.</li>
  <li><strong>SNS</strong> accepts <strong>256 KiB</strong> by default. Since September 2026 a topic can accept up to <strong>1 MiB</strong>
when you raise its <code class="language-plaintext highlighter-rouge">MaximumMessageSize</code> attribute.</li>
</ul>

<p>sqsoverflow therefore offloads at 1 MiB by default, snsoverflow at 256 KiB. Both are configurable
through <code class="language-plaintext highlighter-rouge">payloadSizeThreshold</code>.</p>

<h2 id="usage">Usage</h2>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">dependencies</span> <span class="p">{</span>
    <span class="nf">implementation</span><span class="p">(</span><span class="s">"com.christoph-sens:sqsoverflow:1.1.0"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">s3Client</span> <span class="p">=</span> <span class="nc">S3Client</span><span class="p">.</span><span class="nf">fromEnvironment</span> <span class="p">{</span> <span class="n">region</span> <span class="p">=</span> <span class="s">"eu-central-1"</span> <span class="p">}</span>
<span class="kd">val</span> <span class="py">sqsClient</span> <span class="p">=</span> <span class="nc">SqsClient</span><span class="p">.</span><span class="nf">fromEnvironment</span> <span class="p">{</span> <span class="n">region</span> <span class="p">=</span> <span class="s">"eu-central-1"</span> <span class="p">}</span>

<span class="kd">val</span> <span class="py">client</span> <span class="p">=</span> <span class="nc">SqsExtendedClient</span><span class="p">(</span>
    <span class="n">sqsClient</span><span class="p">,</span>
    <span class="nc">SqsExtendedClientConfig</span><span class="p">(</span><span class="n">payloadStore</span> <span class="p">=</span> <span class="nc">S3BackedPayloadStore</span><span class="p">(</span><span class="n">s3Client</span><span class="p">,</span> <span class="n">bucketName</span> <span class="p">=</span> <span class="s">"my-payload-bucket"</span><span class="p">)),</span>
<span class="p">)</span>

<span class="n">client</span><span class="p">.</span><span class="nf">sendMessage</span><span class="p">(</span><span class="nc">SendMessageRequest</span> <span class="p">{</span> <span class="n">queueUrl</span> <span class="p">=</span> <span class="n">myQueueUrl</span><span class="p">;</span> <span class="n">messageBody</span> <span class="p">=</span> <span class="n">largePayload</span> <span class="p">})</span>

<span class="kd">val</span> <span class="py">messages</span> <span class="p">=</span> <span class="n">client</span><span class="p">.</span><span class="nf">receiveMessage</span><span class="p">(</span><span class="nc">ReceiveMessageRequest</span> <span class="p">{</span> <span class="n">queueUrl</span> <span class="p">=</span> <span class="n">myQueueUrl</span> <span class="p">}).</span><span class="n">messages</span><span class="p">.</span><span class="nf">orEmpty</span><span class="p">()</span>
<span class="k">for</span> <span class="p">(</span><span class="n">message</span> <span class="k">in</span> <span class="n">messages</span><span class="p">)</span> <span class="p">{</span>
    <span class="nf">process</span><span class="p">(</span><span class="n">message</span><span class="p">.</span><span class="n">body</span><span class="p">)</span> <span class="c1">// the original payload, already resolved from S3</span>
    <span class="n">client</span><span class="p">.</span><span class="nf">deleteMessage</span><span class="p">(</span><span class="nc">DeleteMessageRequest</span> <span class="p">{</span> <span class="n">queueUrl</span> <span class="p">=</span> <span class="n">myQueueUrl</span><span class="p">;</span> <span class="n">receiptHandle</span> <span class="p">=</span> <span class="n">message</span><span class="p">.</span><span class="n">receiptHandle</span> <span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">SqsExtendedClient</code> <em>is</em> an <code class="language-plaintext highlighter-rouge">SqsClient</code>, so you can pass it anywhere a plain client is expected.
Messages under the threshold go straight to SQS. Larger ones are written to S3, and SQS only carries
a small JSON pointer plus the <code class="language-plaintext highlighter-rouge">ExtendedPayloadSize</code> message attribute. On receive, the client
fetches the payload and hands you the original body. On delete, it removes the S3 object as well
(<code class="language-plaintext highlighter-rouge">cleanupS3Payload</code>, on by default).</p>

<p>snsoverflow works the same way for <code class="language-plaintext highlighter-rouge">publish</code> and <code class="language-plaintext highlighter-rouge">publishBatch</code>. SNS has no receive side, so the
subscriber resolves the pointer: for an SNS topic that fans out to an SQS queue with raw message
delivery, that’s sqsoverflow’s <code class="language-plaintext highlighter-rouge">SqsExtendedClient</code>.</p>

<h2 id="why-a-port-instead-of-a-wrapper-around-the-java-library">Why a port instead of a wrapper around the Java library</h2>

<h3 id="one-suspend-api-instead-of-sync-and-async-classes">One suspend API instead of sync and async classes</h3>

<p>The Java originals come in two flavors each, for example <code class="language-plaintext highlighter-rouge">AmazonSQSExtendedClient</code> for <code class="language-plaintext highlighter-rouge">SqsClient</code>
and <code class="language-plaintext highlighter-rouge">AmazonSQSExtendedAsyncClient</code> for <code class="language-plaintext highlighter-rouge">SqsAsyncClient</code>. aws-sdk-kotlin clients are <code class="language-plaintext highlighter-rouge">suspend</code>-based
from the start, so one class covers both cases and fits naturally into coroutine code.</p>

<h3 id="interface-delegation-replaces-a-thousand-lines-of-pass-through-code">Interface delegation replaces a thousand lines of pass-through code</h3>

<p>Most of the SQS interface has nothing to do with payload offloading: <code class="language-plaintext highlighter-rouge">createQueue</code>, <code class="language-plaintext highlighter-rouge">listQueues</code>,
<code class="language-plaintext highlighter-rouge">tagQueue</code> and dozens more. The Java library forwards each of these by hand in a base class of
roughly 1,150 lines. Kotlin’s interface delegation does this in one line:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">SqsExtendedClient</span><span class="p">(</span>
    <span class="k">private</span> <span class="kd">val</span> <span class="py">sqsClient</span><span class="p">:</span> <span class="nc">SqsClient</span><span class="p">,</span>
    <span class="k">private</span> <span class="kd">val</span> <span class="py">clientConfig</span><span class="p">:</span> <span class="nc">SqsExtendedClientConfig</span><span class="p">,</span>
<span class="p">)</span> <span class="p">:</span> <span class="nc">SqsClient</span> <span class="k">by</span> <span class="nf">sqsClient</span> <span class="p">{</span>
    <span class="k">override</span> <span class="k">suspend</span> <span class="k">fun</span> <span class="nf">sendMessage</span><span class="p">(</span><span class="n">input</span><span class="p">:</span> <span class="nc">SendMessageRequest</span><span class="p">):</span> <span class="nc">SendMessageResponse</span> <span class="p">{</span> <span class="cm">/* offload if needed */</span> <span class="p">}</span>
    <span class="c1">// ... only the methods with real offload logic are overridden</span>
<span class="p">}</span>
</code></pre></div></div>

<p>sqsoverflow overrides eight methods: <code class="language-plaintext highlighter-rouge">sendMessage</code>, <code class="language-plaintext highlighter-rouge">sendMessageBatch</code>, <code class="language-plaintext highlighter-rouge">receiveMessage</code>,
<code class="language-plaintext highlighter-rouge">deleteMessage</code>, <code class="language-plaintext highlighter-rouge">deleteMessageBatch</code>, <code class="language-plaintext highlighter-rouge">changeMessageVisibility</code>, <code class="language-plaintext highlighter-rouge">changeMessageVisibilityBatch</code> and
<code class="language-plaintext highlighter-rouge">purgeQueue</code>. snsoverflow overrides two: <code class="language-plaintext highlighter-rouge">publish</code> and <code class="language-plaintext highlighter-rouge">publishBatch</code>. Everything else is delegated
to the wrapped client. All three libraries together are about 730 lines of Kotlin, license headers
included.</p>

<h3 id="no-jackson-dependency">No Jackson dependency</h3>

<p>The Java libraries serialize the S3 pointer with Jackson. The ports use kotlinx.serialization, so
nothing in their runtime classpath is Jackson, and the advisories described above do not apply to
them. Fewer transitive dependencies means fewer libraries to keep patched. Dependabot keeps the
remaining dependencies current, and CodeQL and dependency review run on every pull request in all
three repositories.</p>

<h3 id="less-configuration-surface">Less configuration surface</h3>

<p>The original configuration classes carry options for client-side encryption strategies and canned
ACLs. The ports take a <code class="language-plaintext highlighter-rouge">PayloadStore</code> and a handful of behavior flags (<code class="language-plaintext highlighter-rouge">payloadSizeThreshold</code>,
<code class="language-plaintext highlighter-rouge">alwaysThroughS3</code>, <code class="language-plaintext highlighter-rouge">cleanupS3Payload</code>, <code class="language-plaintext highlighter-rouge">ignorePayloadNotFound</code>, <code class="language-plaintext highlighter-rouge">s3KeyPrefix</code>). Encryption is
configured where it belongs: on the bucket, with SSE-S3 or SSE-KMS.</p>

<p><code class="language-plaintext highlighter-rouge">publishBatch</code> in snsoverflow is offload-aware as well. The Java SNS library predates the SNS
<code class="language-plaintext highlighter-rouge">PublishBatch</code> API and only handles <code class="language-plaintext highlighter-rouge">publish</code>.</p>

<h2 id="migrating-from-the-java-libraries">Migrating from the Java libraries</h2>

<p>The important caveat first: <strong>the ports are not wire-compatible with the Java libraries.</strong> The
pointer JSON and the receipt-handle format differ. A message sent by the Java extended client cannot
be resolved by sqsoverflow, and vice versa. Switch all producers and consumers of a queue at the
same time, or drain the queue first.</p>

<p>The <code class="language-plaintext highlighter-rouge">ExtendedPayloadSize</code> attribute name is the same, so SNS-to-SQS fan-out works between
snsoverflow and sqsoverflow.</p>

<p>Before (Java):</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ExtendedClientConfiguration</span> <span class="n">config</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ExtendedClientConfiguration</span><span class="o">()</span>
    <span class="o">.</span><span class="na">withPayloadSupportEnabled</span><span class="o">(</span><span class="n">s3Client</span><span class="o">,</span> <span class="s">"my-payload-bucket"</span><span class="o">);</span>
<span class="nc">SqsClient</span> <span class="n">client</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">AmazonSQSExtendedClient</span><span class="o">(</span><span class="nc">SqsClient</span><span class="o">.</span><span class="na">builder</span><span class="o">().</span><span class="na">build</span><span class="o">(),</span> <span class="n">config</span><span class="o">);</span>
</code></pre></div></div>

<p>After (Kotlin):</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">client</span> <span class="p">=</span> <span class="nc">SqsExtendedClient</span><span class="p">(</span>
    <span class="nc">SqsClient</span><span class="p">.</span><span class="nf">fromEnvironment</span><span class="p">(),</span>
    <span class="nc">SqsExtendedClientConfig</span><span class="p">(</span><span class="n">payloadStore</span> <span class="p">=</span> <span class="nc">S3BackedPayloadStore</span><span class="p">(</span><span class="n">s3Client</span><span class="p">,</span> <span class="n">bucketName</span> <span class="p">=</span> <span class="s">"my-payload-bucket"</span><span class="p">)),</span>
<span class="p">)</span>
</code></pre></div></div>

<p>Options without an equivalent:</p>

<table>
  <thead>
    <tr>
      <th>Java option</th>
      <th>In the ports</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Client-side encryption (<code class="language-plaintext highlighter-rouge">ServerSideEncryptionStrategy</code>)</td>
      <td>Configure SSE-S3 or SSE-KMS on the bucket</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ObjectCannedACL</code></td>
      <td>Use bucket policies</td>
    </tr>
    <tr>
      <td>SNS per-message <code class="language-plaintext highlighter-rouge">"S3Key"</code> attribute</td>
      <td>Use <code class="language-plaintext highlighter-rouge">s3KeyPrefix</code> in the config</td>
    </tr>
    <tr>
      <td>Legacy <code class="language-plaintext highlighter-rouge">SQSLargePayloadSize</code> attribute name</td>
      <td>Always <code class="language-plaintext highlighter-rouge">ExtendedPayloadSize</code></td>
    </tr>
  </tbody>
</table>

<h2 id="verifying-what-you-download">Verifying what you download</h2>

<p>Every release is built and published by a GitHub Actions workflow. The jars, POM and Gradle module
metadata on Maven Central carry a signed
<a href="https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations">build provenance attestation</a>
that ties each file to the repository, the workflow and the tagged commit it was built from. You can
check a downloaded jar with the GitHub CLI:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gh attestation verify sqsoverflow-1.1.0.jar <span class="nt">--repo</span> christoph-sens/sqsoverflow
</code></pre></div></div>

<h2 id="licensing">Licensing</h2>

<p>sqsoverflow and snsoverflow are derivative works of the AWS libraries under Apache-2.0, with
per-file attribution and a NOTICE file listing what was changed. s3overflow is an independent
reimplementation that takes no code from <code class="language-plaintext highlighter-rouge">payload-offloading-java-common-lib-for-aws</code>.</p>

<h2 id="try-it">Try it</h2>

<ul>
  <li>s3overflow: <a href="https://github.com/christoph-sens/s3overflow">https://github.com/christoph-sens/s3overflow</a></li>
  <li>sqsoverflow: <a href="https://github.com/christoph-sens/sqsoverflow">https://github.com/christoph-sens/sqsoverflow</a></li>
  <li>snsoverflow: <a href="https://github.com/christoph-sens/snsoverflow">https://github.com/christoph-sens/snsoverflow</a></li>
</ul>

<p>Issues and pull requests are welcome. If you run into a case the libraries don’t cover yet, such as
another AWS service that could use the same pattern, open an issue.</p>

<p><em>I’m a freelance developer and consultant working on Kotlin, Java and AWS. The story of how these
libraries came about is on my website (in German):
<a href="https://www.christoph-sens.com/post/der-scan-befund-der-nie-verschwindet-wie-eine-ki-drei-aws-legacy-bibliotheken-an-einem-nachmittag">1150 Zeilen Legacy, eine Zeile Kotlin</a>.</em></p>]]></content><author><name>Christoph Sens</name></author><category term="kotlin" /><category term="aws" /><category term="sqs" /><category term="sns" /><category term="s3" /><category term="coroutines" /><summary type="html"><![CDATA[The AWS Java extended client libraries for SQS and SNS pin old Jackson versions with known advisories. sqsoverflow and snsoverflow port them to aws-sdk-kotlin and coroutines, without Jackson.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://christoph-sens.github.io/assets/images/overflow-social-preview.png" /><media:content medium="image" url="https://christoph-sens.github.io/assets/images/overflow-social-preview.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>