Nessys (Nuclear Envelop Segmentation System) is a module for the automated segmentation of nuclei in fluorescence images. It can handle 2D, 3D, and time lapse datasets.
Important: Nessys uses the signal of the nuclear envelope in order to identify nuclei. With such labeling, Nessys is particularly well suited for dense, heterogeneous and overlapping nuclei. Note also that although Nessys has not been designed for Dapi or Hoescht stained nuclei, it may still give satisfactory results when nuclei are not too densely packed or not too textured.

Notes on Image Quality

Image quality is critical to get right for image quantification purpose. It is essential to properly adjust image acquisition parameters with the analysis application that needs to be performed in mind. The choices of acquisition parameters depend on the balance between image disk size, acquisition time, sample viability, photo-bleaching and the required image quality.

Below is a list of image properties that are strongly recommended for Nessys segmentation :

Walk-through video tutorial

The following video provides a step by step guide on how to segment the sample image available for download on our owncloud server (password is ‘nessys’).

How to adjust parameters

The following sections describe parameters for each individual steps in the Nessys pipeline and provides advice on how to properly adjust parameters for a new set of images. It assumes that you have watched the video tutorial above and/or that you are running Nessys as you follow this guide.


An optional denoising step has recently been added to the segmentation pipeline.
This step is useful if the signal to noise ratio of the image is low. For example, applying a Gaussian filter to the image below may help the segmentation:

denoising image

NB1: Gaussian blur does not preserve edges very well so when applying this filter, use radii that are lower than 1. For the above image a value of 0.7 was used in all directions.

NB2: If the image is of good quality, it is preferable to skip this step by selecting ‘skip’ in the ‘Available methods’ dropdown list.

Ridge Enhancement

This step consists in smoothing and enhancing nuclear contours. A complete description of the algorithm used for this step and of the parameters effects can be found in (Jacob and Unser, 2004).

Example of expected results at this stage:
Note how the noise in the image becomes apparent as tiny ridglets when the scale is too low (scale = 0.8). When the scale is too high (scale = 4), the contour of the nuclei look blurred and widened. In this case a scale of 1.5 gives a good compromise.

steered image

Jacob, M., and Unser, M. (2004). Design of steerable filters for feature detection using canny-like criteria. IEEE Trans. Pattern Anal. Mach. Intell. 26, 1007–1019.

Slice by slice segmentation

This step consists in identifying cell areas on each z-slice of the image. This can be divided into 3 sub-tasks:

  1. Identify intensity maxima in the image. These maxima are used as the ‘origins’ for tracing the contour of individual nuclei.
  2. Trace the contour of nuclei : Staring from a maximum, this process follows the ridges in the vicinity to create a ‘trace’.
  3. Rank shapes : Once a trace is constructed, multiple shapes maybe defined from it. At this step, the user trains a classifier to define which shape defines a valid nucleus.

Maxima Detection

This step consists in identifying locations in the image where bright ridges can be found. As mentioned above, these maxima are then used as the ‘origins’ for tracing the contour of individual nuclei. Therefore, the number of maxima will dictate how many attempts will be done to identify a new shape in the image. The higher the number of maxima, the longer the segmentation time but the more likely all the nuclei will be found.

Maxima Threshold : To adjust this threshold, find the dimmest cells and try to obtain 2-3 maxima per cell (located on the ridge of the nucleus). Increase the threshold when too many maxima are detected in the background, decrease the threshold when some cells do not have maxima.

Contour Detection

This step (called ‘Ridge Segmentation’ in the interface) is the step which takes the maxima identified previously and attempts to trace the contour of individual nuclei.

Adjust the parameters in the following order:

Shape ranking

This step consists in telling Nessys which detected shapes are indeed nuclei and which shapes are incorrect (space between nuclei or half of a nucleus for example). In other words this step makes you train a classifier so that wrong shapes are eliminated from the final result.

A classifier is provided by default (E70_Classifier), however it is often required to train a new classifier to obtain optimal results.

  1. If you have not generated shapes with the default classifier, click on ‘Generate Shapes’. The generated shapes will depend on the contour detection parameters used (see above). This will draw shapes in the ‘annotation’ channel which can then be selected (right click) to train the classifier.
  2. To train the classifier, you need to select at least 10 shapes for each class (valid and invalid). Once 10 objects of each have been selected, you can click on ‘Create Best Guess’ in order to preview what the final result will look like.
  3. After checking that the new classifier gives satisfactory results over several z-sections, you can save the classifier to file for future use.

3D Linkage

This step consists in linking the 2D areas identified in the previous step in order to obtain 3D volumes. Nessys will provide default values based on the parameters provided in the previous steps.

Basic parameters
Advanced parameters

The 3D linkage step works by first creating a graph of potential linkages across z-sections. Each potential link is assigned with a ‘likeliness’ score which depends on the Jaccard Index (index of similarity). Once the graph is constructed. The links are ranked in descending order of their score and assigned a unique volume id. The volume id is then propagated when two ids meet, provided that the merged volume remains within bounds.
The following parameters have an impact on the creation of the graph of potential links :

There exists an additional subtlety to take into account regarding the creation of the graph of potential connections. Nessys attempts to correct for errors which occured in the previous step. Area may represent merged cells, in order to correct for this type of error, the graph creation step must allow potential links to be created between properly identified areas in one slice and merged area in the subsequent slice. The centroid distance may be relatively large in such a case. Increasing the search distance can lead to the counter-intuitive result of properly splitting an area which was a merge in the previous step. Thus in order to properly adjust these parameters one has to keep a look at the inputs of this step and check if a lot of splits or merges are present.

Images larger than 4GB

If you are using the stand alone Nessys application and you want to segment images larger than 4GB, please refer to instructions in the README.txt file located in the root folder of the Nessys installation.