A macOS application to compute image differences using Metal shaders
Tweet
Mantis Shrimp Manual
Table of Contents
Basic usage
Mantis Shrimp is a document-based app that computes differences between images using Metal shaders.
To start a new project simply go to File → New, or press ⌘ N. Then, simply drag & drop the 2 images that you want to compare into the document window.
By default, the Image Diff will show a linear RGB difference between images A and B. The resolution and color space of the Image Diff is the same as Image A. If Image B is of a different aspect, it will be stretched to match Image A.
You can save the project with File → Save, and you can export the Image Diff to PNG with File → Export As....
There are 2 vertical separators between the different sections, so you can scale the different areas separately. Also note that in the editor window, you can press command + + and command + - to make the font bigger or smaller, respectively.
Options
There are a few options that let you modify the output:
- Linear RGB. When this flag is set, the image will be converted to linear RGB so the differences will be computed in linear space. In the output image, the gamma is applied back automatically. This automatic conversion only works for 8-bit color images, and 8-bit gray images where the hardware supports it (e.g. Macbook Pros from 2017 or earlier do not support the automatic conversion). For 16-bit or 32-bit images, the flag is ignored.
- Clear color. This is the color that the output canvas will be cleared with when writing your own shader. See Custom Shaders below.
- Scale. This is a factor that gets multiplied to the output to make it brighter. You can access this parameter in your Custom Shader for your own purposes.
- Object Grid. This is the number of object thread groups in Mesh shaders. The maximum value is
32×32. - Mesh Grid. This is the mesh grid per object thread group in a mesh shader. The maximum value is
5×5. - is3D. When enabled, back-face culling will be enabled and a 32-bit float depth buffer will be set. This is only useful when creating Mesh shaders.
Shader Presets
Under the Presets menu you can find a few shader presets that can help you visualize different types of image differences. Here's a visual summary:
- RGB Diff. This is the default image difference, where the absolute difference of each color channel is displayed.
- Alpha Diff. This compares the alpha channel of two images. It will appear white where the alpha is different, and black otherwise.
- Perceived lightness. This computes the difference from the perceived lightness between Image A and Image B, where the Image Diff will appear yellow where Image B lightness is greater, and blue where Image A lightness is greater. If they are the same, it should appear black. This assumes that Linear RGB is selected.
- Side-by-side. This splits the output vertically in two and shows Image A to the left, and Image B to the right, with a slight alpha-blend. Use the Scale parameter to move the vertical line from left to write. If you want to change the blend factor, edit the
blendvariable in the code, where1would make it fully opaque. - Signed Alpha + RGB. This compares both the alpha channel and the color of two images. Where red, Image A is opaque but Image B is transparent. Where green, the opposite is true. Where both images are opaque, the intensity of blue represent how different colors are, being black when there is no difference.
- Pass-through. The output will be the same as Image A. This is mostly a placeholder for starting to write your custom shader. See next section.
- HDR Threshold. This shader only needs Image A as input. It can be used to visualize which parts of the image are greater than 1 in an HDR image. This can be useful to debug if a light in an environment map is really bright, or if it has been clamped. The Scale parameter can be used to increase the brightness threshold.
- SDF Animation. This shader is an example of animation, to illustrate that Mantis Shrimp can also be used as a "shader toy". It doesn't require any image, but if Image A exists, its colors will be used as input. The Scale parameter can be used to control the intensity of the circles.
Custom shaders
You can write your own fragment shaders in Mantis Shrimp using Metal Shading Language.
After writing your code, press the "play" button ▶️ to build and run your shader. There's also a record button, the red circle, to record an mp4 video with the output of your shader, in case it's animated. Notice that the record button is grayed out and disabled while recording. You can select the number of seconds to record and the framerate (15, 20, 30, or 60 fps).
There are a few predefined keywords to make it easier to write a shader. Start with any of the presets to get an example. The easiest one is the Pass-through shader:
fragment half4 main()
{
float4 a = texA.sample(sam, frag.uv);
return half4(a);
}
The first line is an alias for the actual fragment shader header:
fragment half4 diffFragment(VertexInOut frag [[stage_in]], texture2dtexA [[ texture(0) ]], texture2d texB [[ texture(1) ]], sampler sam [[ sampler(0) ]], constant Uniforms& uni [[ buffer(0) ]])
The fragment input frag is a single quad that represents the whole canvas, and texA and texB are the input images. When there is no image, a default white image is assigned. The default sampler sam is a point sampler where the address mode is set to clamp to edge. And the Uniforms are common variables passed to the shader. See the full definition below:
struct VertexInOut {
float4 position [[position]];
float4 color; // unused at the moment
float2 uv;
};
struct Uniforms {
float time; // the running time in seconds
float scale; // a value between 0 and 10, passed by the UI
float2 touch; // UV coordinate of the last point you clicked
float2 resolution; // (width, height) in pixels
float2 pixelSize; // (1/width, 1/height)
float2 resolutionB; // (width, height) for texture B
float2 pixelSizeB; // (1/width, 1/height) for texture B
uint4 gridSize; // Mesh grid size for mesh shaders
};
float3 linear_to_gamma(float3 color);
float3 gamma_to_linear(float3 color);
float3 srgb_to_p3(float3 color);
float3 p3_to_srgb(float3 color);
There are a couple of functions to convert between linear and gamma RGB. This is done automatically if you select Linear RGB, but if the input image is 16-bit or 32-bit, you will need to call these functions manually if you want to convert to linear RGB.
There are a couple of conversions between sRGB and Display-P3 color spaces. You can read about the conversion and matrices I use in this blog post: Exploring the display-P3 color space. Note that the conversions assume that the color is in linear RGB color space. If you load a 16-bit PNG in display-P3, you will need to apply the conversions yourself. For instance, the example below removes all colors (sets the alpha to 0) that are NOT outside the sRGB gamut (so it keeps only the very bright colors that fall outside the sRGB gamut).
Here are the 16-bit display-P3 images for testing and the code:
fragment half4 main()
{
float4 a = texA.sample(sam, frag.uv);
// filter to keep only the colours that are outside the sRGB spectrum
float3 c = gamma_to_linear(a.rgb);
c = p3_to_srgb(c);
float3 negMask = step(c, -0.001);
float3 posMask = step(1.001, c);
float3 mask = step(1.0, negMask + posMask);
float p3Mask = step(1.0, mask.r + mask.g + mask.b);
c = srgb_to_p3(c);
c = linear_to_gamma(c);
float4 out = float4(c, p3Mask);
return half4(out);
}
Pixel format
The pixel format of the inputs depends on the image format and bit depth. When you drag & drop an image in Mantis Shrimp, the pixel format will be displayed below the image. For 8-bit RGBA images, usually the format is bgra8Unorm, or bgra8Unorm_srgb. Whether the sRGB flag (the "_srgb") is set or not is irrelevant, since internally Mantis Shrimp reinterprets the image format based of the Linear RGB flag.
As mentioned earlier, the output is configure to match Image A. That means it will be the same size, the same color space, and a compatible pixel format. Below the output canvas you can see the output format. The Linear RGB flag will change the format of the output where possible, so 8-bit formats will end with "_srgb" when the flag is set. Because the output is used for exporting the image to disk, usually the channels will be in RGBA order (just because it's easier to serialize).
The display canvas is in a fixed format, compatible with display-P3 color space. Where the hardware supports it, the view is in bgra10_xr, but on older Macs that will be bgr10a2Unorm instead. The final gamma conversion is done behind the scenes inside a shader, where necessary.
Mesh shaders
It is possible to create more than pixels with Mantis Shrimp. If your Mac supports Metal 3, it is possible to create geometry as well using Mesh Shaders. To learn more about mesh shaders, watch the WWDC2022 talk to learn more about mesh shaders. The basic summary is that a mesh shader is split in an Object Stage and a Mesh Stage. Both of them are programmable, and they are similar to compute kernels. As compute kernels, they are launched in grids of thread groups.
From version 1.1 of Mantis Shrimp, there are some options to control the grid size of the object stage, the Object Grid, as well as the Mesh Grid per object thread group. The maximum Object Grid size is 32×32, whereas the maximum mesh grid size is 5×5. In the object and mesh shaders you can check the grid size with uni.gridSize, where xy correspond to the width and height of the object grid, and zw contain the mesh grid size. In the shader you have the thread index to figure out which thread is being processed.
Before going into details of the object and mesh stages, let's see an example of the fragment shader. When you want to use mesh shaders, you will need to use a different fragment function with a slightly different syntax. Mantis Shrimp will switch to use Mesh Shaders when that fragment function is defined. This is a simple fragment function that computes a rough image differential:
fragment float4 fragmentMesh()
{
float4 quadColor = texA.sample(sam, in.v.uv.xy);
float4 diff = abs(in.p.color - quadColor);
return float4(diff.rgb, 1.0);
}
Notice that the fragment function needs to be named fragmentMesh, not main, and that the inputs are in a structure called in, defined as:
struct VertexOut
{
float4 position [[position]];
float3 normal;
float4 uv;
};
// Per-vertex primitive data.
struct PrimOut
{
float4 color;
};
struct FragmentIn
{
VertexOut v;
PrimOut p;
};
The reason that the function above computes a differential is because by default the object and mesh shaders create a series of quads where each quad color is sampled from the center of that quad in texture A. So the fragment shader above compute the difference of each pixel to the center of the quad it belongs to.
For reference, the full signature of the fragment function is:
fragment float4 fragmentMesh(FragmentIn in [[stage_in]],
texture2d texA [[ texture(0) ]],
texture2d texB [[ texture(1) ]],
sampler sam [[ sampler(0) ]],
constant Uniforms& uni [[ buffer(0) ]])
If you want to write your own object and mesh stages, you need to write these functions:
void objectStage(); void meshStage();
The actual signature of those functions is:
[[object]]
void objectStage(object_data MeshPayload& payload [[payload]],
mesh_grid_properties props,
constant Uniforms& uni [[ buffer(0) ]],
uint3 positionInGrid [[threadgroup_position_in_grid]]);
[[mesh]]
void meshStage(TriangleMeshType output,
const object_data MeshPayload& payload [[payload]],
texture2d texA [[ texture(0) ]],
texture2d texB [[ texture(1) ]],
sampler sam [[ sampler(0) ]],
constant Uniforms& uni [[ buffer(0) ]],
uint lid [[thread_index_in_threadgroup]],
uint tid [[threadgroup_position_in_grid]])
where MeshPayload and TriangleMeshType are defined as:
struct Vertex
{
float4 position;
float4 normal;
float2 uv;
};
static constexpr constant uint32_t MaxVertexCount = 128;
static constexpr constant uint32_t MaxPrimitiveCount = 128;
using TriangleMeshType = metal::mesh< VertexOut, PrimOut, MaxVertexCount, MaxPrimitiveCount, topology::triangle >;
struct MeshPayload
{
Vertex vertices[MaxVertexCount];
float4x4 transform;
float2 uv;
float2 size;
uint32_t primitiveCount;
uint8_t vertexCount;
};
Note that you can't overwrite the signatures of the functions, but you can overwrite the structs and the TriangleMeshType 😃 Be sure you know what you are doing, though!
In MantisShrimpExamples you have several examples of Mesh Shaders. For instance, #genuary10 cubes is an example that outputs a single point per object, positioned in a regular grid. In the mesh stage, each point is transformed into a cube, that is, 6 faces, or 12 triangles. Note that the back faces may be rendered on top of the front faces, or that the depth will be ignored by default. The is3D flag in the options shown above will enable back-face culling (front-facing triangles are CCW) and set a depth buffer, so can create basic 3D scenes with these. See the output from that example below: