Extracting features from deeplearning4j layer

Issue

I am trying to extract the layer activations to save them locally as features. I'm still new to CNNs so I'd like to show what I did and I'd like to know if what I'm doing is correct:

public static void main(String[] args) throws IOException {
    ComputationGraph vgg16transfer = getComputationGraph();

    for (File file : new File(ImageClassifier.class.getClassLoader().getResource("mydirectory").getFile()).listFiles()) {
        Map<String, INDArray> stringINDArrayMap = extractTwo(file, vgg16transfer);
        //Extract the features from the last fully connected layers
        saveCompressed(file,stringINDArrayMap.get("fc2"));
    }
}

/**
 * Retrieves the VGG16 computation graph
 * @return ComputationGraph from the pretrained VGG16
 * @throws IOException
 */
public static ComputationGraph getComputationGraph() throws IOException {
    ZooModel zooModel = new VGG16();
    return (ComputationGraph) zooModel.initPretrained(PretrainedType.IMAGENET);
}

/**
 * Compresses the input INDArray and writes it to file
 * @param imageFile the original image file
 * @param array INDArray to be saved (features)
 * @throws IOException
 */
private static void saveCompressed(File imageFile, INDArray array) throws IOException {
    INDArray compress = BasicNDArrayCompressor.getInstance().compress(array);
    Nd4j.write(compress,new DataOutputStream(new FileOutputStream(new File("features/" + imageFile.getName()+ "feat"))));
}

/**
 * Given an input image and a ComputationGraph it calls the feedForward method after rescaling the image.
 * @param imageFile the image whose features need to be extracted 
 * @param vgg16 the ComputationGraph to be used.
 * @return a map of activations for each layer
 * @throws IOException
 */
public static Map<String, INDArray> extractTwo(File imageFile, ComputationGraph vgg16) throws IOException {
    // Convert file to INDArray
    NativeImageLoader loader = new NativeImageLoader(224, 224, 3);
    INDArray image = loader.asMatrix(imageFile);

    // Mean subtraction pre-processing step for VGG
    DataNormalization scaler = new VGG16ImagePreProcessor();
    scaler.transform(image);

    //Call the feedForward method to get a map of activations for each layer
    return vgg16.feedForward(image, false);
}

So basically I am calling the feedForward method and obtaining the activations from the fc2 layer.

I have several questions about this:

1) Is the code I wrote indeed extracting features that can be saved and stored for further usages?

2) How would I go about doing PCA/Whitening on the extracted features?

3) Is there any way I could encode this to VLAD as proposed but such paper: https://arxiv.org/pdf/1707.00058.pdf

4) I then would like to compare the saved features, I did so using a simple Euclidean distance and it seems to be working although results are not the best. Is there some kind of pre-processing I should do or are the saved features directly comparable?

Thanks.

Solution

Edit: new link for transfer learning is here: https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/features/transferlearning/README.md

Note that this message is 4 years old and the repositories have all moved to a mono repo here: https://github.com/eclipse/deeplearning4j

For the extracted features, Nd4j has a PCA you can use: https://github.com/deeplearning4j/nd4j/blob/master/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/dimensionalityreduction/PCA.java

That's kind of redundant though. I would consider just directly using transfer learning instead. You don't have to do any of this manually. See the docs here for more examples: https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-spark-examples/dl4j-spark/src/main/java/org/deeplearning4j/transferlearning/vgg16/TransferLearning.md

The transfer learning api will give you what you need for using pretrained models while modifying them only to use a different output layer. We even cover this with VGG16 right in the example above.

Answered By - Adam Gibson