I am working on a project that requires appending to a AES/CTR encrypted file. Now, since it is counter mode I know that I can advance the counter to any location and start reading at the location in the file. What I am wondering though is if there is a way for me to fetch the current IV that Cipher has access to after it has been used.

Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
SecretKeySpec keySpec = new SecretKeySpec(aeskey, "AES");
IvParameterSpec ivSpec = new IvParameterSpec(iv);

c.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);

CipherOutputStream cipher_out = new CipherOutputStream(output, c);

try {
    while (true) {
        cipher_out.write(input.readByte());
    }
} catch (EOFException e) {
}

byte curIV[] = c.getIV();

Instead I am finding that curIV rather than having the updated IV has the same IV that I passed into the ivSpec to begin with. Is there no way to get the current IV?

The idea is to store:

开发者_如何转开发<AES key><begin IV><current IV>

in a asymmetricly encrypted file, this way that can be decrypted, read and we can start reading the AES encrypted file from the beginning or we can append new data to our output file using the <current IV> that we have stored.

Any other suggestions on how to implement this?

Java according to documentation I have found uses the following (close to RFC3686):

<NONCE><COUNTER>

As its input into the CTR, and to update the counter it is considered to be a big endian number.

This is provided as the IvParameterSpec seen above.

Besides the point, what I am trying to get back is the counter, whether we want to call that the IV, or if we want to call it the counter, or the nonce + iv + counter.

More information about Suns implementation of CTR: http://javamex.ning.com/forum/topics/questions-on-aes-ctr-mode

In experimenting with the "SunJCE" provider, AES in CTR-mode follows the proposal published by NIST, where an initial counter value is simply incremented by one with each successive block. This is consistent with the general guidance given in NIST SP 800‑38A, Appendix B.1., when the number of incremented bits, m, is the number of bits in the block, b.

This is contrary to RFC 3686. That is, the entire counter is incremented, not just a limited portion as specified in RFC 3686.

You can know the block index by counting blocks (starting with zero), or by measuring the length of the cipher text and performing integer division by the block size. If those options seem too easy, you can also XOR the last block of cipher text with the corresponding plain text, decrypt that result, and subtract the IV to yield the block index.

To append, simply set the IV to the original IV plus the block index. If you are writing streams that can end with a partial block, you'll have some extra work to do to get the stream into the correct state.

int BLOCK_SIZE = 16;
BigInteger MODULUS = BigInteger.ONE.shiftLeft(BLOCK_SIZE * 8);
...
/* Retrieve original IV. */
byte[] iv = ... ;

/* Compute the index of the block to which data will be appended. */
BigInteger block = BigInteger.valueOf(file.length() / BLOCK_SIZE);
/* Add the block to the nonce to find the current counter. */
BigInteger nonce = new BigInteger(1, iv);
byte[] tmp = nonce.add(block).mod(MODULUS).toByteArray();
/* Right-justify the counter value in a block-sized array. */
byte[] ctr = new byte[BLOCK_SIZE];
System.arraycopy(tmp, 0, ctr, BLOCK_SIZE - tmp.length, tmp.length);
/* Use this to initialize the appending cipher. */
IvParameterSpec param = new IvParameterSpec(ctr);

(Bad form, I know, but answer my own question to document it for other people)

The updating of a counter ...

public static byte[] update_iv(byte iv[], long blocks) {
    ByteBuffer buf = ByteBuffer.wrap(iv);
    buf.order(ByteOrder.BIG_ENDIAN);
    long tblocks = buf.getLong(8);
    tblocks += blocks;
    buf.putLong(8, tblocks);

    return buf.array();
}

Explanation of AES/CTR-BE

This is the basic idea. If you read erickson's answer to my question you will see that the IV is basically:

<8 bytes nonce><8 bytes counter>

The counter is stored in BIG_ENDIAN format, so that if you were to pull out the counter at state 1 you'd get this:

0x0 0x0 0x0 0x1

Then when it gets to the second block it updates it to

0x0 0x0 0x0 0x2

and so forth, it can technically overflow into the nonce, but it is not suggested to encrypt that much data in the first place.

Now personally I create the nonce/counter randomly. So that it becomes even harder to guess, this is not a requirement.

What the above does is allow you to update the counter with how many blocks into the counter you want to go, it doesn't matter whether you start at 0x1 or any other counter value (random like myself).

Now, if we end on half a block or less we need to make sure we move forward in the AES-CTR for a couple of bytes, so we can simply do:

c.update(new byte[count])

where count is the amount of characters that is the distance into the block.

My implementation explained

The way I have my keyfile stored on disk (in plaintext, PLEASE DO NOT DO THIS!) is as follows:

<16 bytes AES key>
<8 bytes nonce>
<8 bytes counter>
<8 bytes (long) block count>
<4 byte partial block count>

This gives us all the information we need to append something to the end of an already encrypted file without having to first decrypt any content. Which is absolutely fantastic for log files that need to be encrypted, as well any other content that can be streamed.

Testing

The way I tested that this actually worked is as follows:

echo "1234567890ABCDEF" > file1
echo "0987654321ABCDEFGHIJKLMNOPQRSTUVWXYZ" > file2
cat file1 file2 > file3

Now, if we encrypt file1 and then append file2 we should get the same output as when we encrypt file3 so long as we use the same key/IV for both.

javac AESTest.java # Compile the java file
java AESTest key file1 append.aes
java AESTest key file2 append.aes append 

Adding append tells the program to go into append mode and move the block count forward and go partially into the next CTR cycle using the aforementioned c.update() method. From there on it starts encrypting like any other time, and simply appends the data to the output file.

java AESTest key file3 noappend.aes

Since my program will simply ignore the block count/partial block count unless you pass in the argument append this will simply start encrypting the file using the same key/IV as before.

Now if we look at both files using a HexEditor or vbindiff we can verify that the two files are exactly the same, yet one had content appended to it after the fact.

Full source ...

(Please do note that this is the first time I programmed in Java since high school, which was a few years ago, please excuse the horrible code)

Full source code for my program where all of this is implemented.

import java.util.Random;
import java.security.*;
import javax.crypto.*;
import javax.crypto.spec.*;
import java.lang.String;
import java.io.File;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

public class AESTest {

    public static byte[] update_iv(byte iv[], long blocks) {
        ByteBuffer buf = ByteBuffer.wrap(iv);
        buf.order(ByteOrder.BIG_ENDIAN);
        long tblocks = buf.getLong(8);
        tblocks += blocks;
        buf.putLong(8, tblocks);

        return buf.array();
    }

    public static void main(String args[]) throws Exception {
        if (args.length < 3) {
            System.out.println("Not enough parameters:");
            System.out.println("keyfile input output [append]");
            return;
        }


        File keyfile = new File(args[0]);
        DataInputStream key_in;
        DataOutputStream key_out;
        DataInputStream input = new DataInputStream(new FileInputStream(args[1]));
        DataOutputStream output = null;

        byte key[] = new byte[16 + 16];
        byte aeskey[] = new byte[16];
        byte iv[] = new byte[16];
        byte ivOrig[] = new byte[16];
        long blocks = 0;
        int count = 0;

        if (!keyfile.isFile()) {
            System.out.println("Creating new key");
            Random ranGen = new SecureRandom();
            ranGen.nextBytes(aeskey);
            ranGen.nextBytes(iv);

            iv = update_iv(iv, 0);

            System.arraycopy(iv, 0, ivOrig, 0, 16);
       } else {
            System.out.println("Using existing key...");
            key_in = new DataInputStream(new FileInputStream(keyfile));

            try {
                for (int i = 0; i < key.length; i++)
                    key[i] = key_in.readByte();
            } catch (EOFException e) {
            }

            System.arraycopy(key, 0, aeskey, 0, 16);
            System.arraycopy(key, 16, iv, 0, 16);
            System.arraycopy(key, 16, ivOrig, 0, 16);

            if (args.length == 4) {
                if (args[3].compareTo("append") == 0) {
                    blocks = key_in.readLong();
                    count = key_in.readInt();

                    System.out.println("Moving IV " + blocks + " forward");
                    iv = update_iv(iv, blocks);
                    output = new DataOutputStream(new FileOutputStream(args[2], true)); // Open file in append mode
                }
            }
        }

        if (output == null)
            output = new DataOutputStream(new FileOutputStream(args[2])); // Open file at the beginnging

        key_out = new DataOutputStream(new FileOutputStream(keyfile));

        Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
        SecretKeySpec keySpec = new SecretKeySpec(aeskey, "AES");
        IvParameterSpec ivSpec = new IvParameterSpec(iv);
        c.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);

        if (count != 0) {
            c.update(new byte[count]);
        }

        byte cc[] = new byte[1];
        try {
            while (true) {
                cc[0] = input.readByte();
                cc = c.update(cc);
                output.writeByte(cc[0]);

                if (count == 15) {
                    blocks++;
                    count = 0;
                } else {
                    count++;
                }
            }
        } catch (EOFException e) {
        }

        cc = c.doFinal();
        if (cc.length != 0)
            output.writeByte(cc[0]);

        // Before we quit, lets write our AES key, start IV, and current IV to disk
        for (int i = 0; i < aeskey.length; i++)
            key_out.writeByte(aeskey[i]);

        for (int i = 0; i < ivOrig.length; i++)
            key_out.writeByte(ivOrig[i]);


        System.out.println("Blocks: " + blocks);
        System.out.println("Extra: " + count);
        key_out.writeLong(blocks);
        key_out.writeInt(count);

    }
}

You are right in that getIV returns the original IV, not the current one after some encryption/decryption has taken place.

In Java, the 16 bytes passed to the AES block cipher in CTR mode are the IV plus the current block number (added as if both were 16-byte bignums in big-endian format, see code below).

Be sure you read this StackOverflow post, it has lots of good advice for avoiding security pitfalls of CTR mode (summary: NEVER encrypt twice with the same IV).

For your use case, you just need to store the beginning IV plus the block number (and not even the block number if you can get the file size some other way). You can compute the current IV from that for either further encryption or random seek decryption.

Code to compute the correct IV to use given the block number (the first block being 0):

int block = ...;
byte[] iv = ...;
byte[] blockbytes = new byte[16];
for (int i = 0; i < 4; i++) blockbytes[15 - i] = (byte)(block >> 8*i);
int carry = 0;
for (int i = 15; i >= 0; i--) {
  int sum = (iv[i] & 255) + (blockbytes[i] & 255) + carry;
  iv[i] = (byte)sum;
  carry = sum >> 8;
}

Caveat: I got this from figuring out what the code does - I didn't see it in the spec so it is possible the algorithm varies with provider.

Here's a more complete test program you can try:

import java.math.*;
import javax.crypto.*;
import javax.crypto.spec.*;
public class TestCTR {
  static SecretKeySpec keySpec = new SecretKeySpec(new BigInteger("112233445566778899aabbccddeeff00", 16).toByteArray(), "AES");
  static IvParameterSpec ivSpec = new IvParameterSpec(new BigInteger("66778899aaffffffffffffffffffffff", 16).toByteArray());

  public static void main(String[] args) throws Exception {
    byte[] plaintext = new byte[256];
    for (int i = 0; i < 256; i++) plaintext[i] = (byte)i;

    // encrypt with CTR mode                                                                                                           
    byte[] ciphertext = new byte[256];
    Cipher c = Cipher.getInstance("AES/CTR/NoPadding");
    c.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);
    c.doFinal(plaintext, 0, 256, ciphertext, 0);

    // decrypt, implementing CTR mode ourselves                                                                                        
    Cipher b = Cipher.getInstance("AES/ECB/NoPadding");
    b.init(Cipher.ENCRYPT_MODE, keySpec);
    for (int block = 0; block < 16; block++) {
      byte[] iv = ivSpec.getIV();
      int carry = 0;
      byte[] blockbytes = new byte[16];
      for (int i = 0; i < 4; i++) blockbytes[15 - i] = (byte)(block >> 8*i);
      for (int i = 15; i >= 0; i--) {
        int sum = (iv[i] & 255) + (blockbytes[i] & 255) + carry;
        iv[i] = (byte)sum;
        carry = sum >> 8;
      }
      b.doFinal(iv, 0, 16, iv, 0);
      for (int i = 0; i < 16; i++) plaintext[block*16+i] = (byte)(ciphertext[block*16+i] ^ iv[i]);
    }

    // check it                                                                                                                        
    for(int i = 0; i < 256; i++) assert plaintext[i] == (byte)i;
  }
}