NIPs nostr improvement proposals

NIP-44 - Encrypted Payloads (Versioned)

Table of Contents

Encrypted Payloads (Versioned)

optional

The NIP introduces a new data format for keypair-based encryption. This NIP is versioned to allow multiple algorithm choices to exist simultaneously. This format may be used for many things, but MUST be used in the context of a signed event as described in NIP-01.

Note: this format DOES NOT define any kinds related to a new direct messaging standard, only the encryption required to define one. It SHOULD NOT be used as a drop-in replacement for NIP-04 payloads.

Versions

Currently defined encryption algorithms:

Limitations

Every nostr user has their own public key, which solves key distribution problems present in other solutions. However, nostr's relay-based architecture makes it difficult to implement more robust private messaging protocols with things like metadata hiding, forward secrecy, and post compromise secrecy.

The goal of this NIP is to have a simple way to encrypt payloads used in the context of a signed event. When applying this NIP to any use case, it's important to keep in mind your users' threat model and this NIP's limitations. For high-risk situations, users should chat in specialized E2EE messaging software and limit use of nostr to exchanging contacts.

On its own, messages sent using this scheme have a number of important shortcomings:

Lack of forward secrecy may be partially mitigated by only sending messages to trusted relays, and asking relays to delete stored messages after a certain duration has elapsed.

Version 2

NIP-44 version 2 has the following design characteristics:

Encryption

  1. Calculate a conversation key
    • Execute ECDH (scalar multiplication) of public key B by private key A Output shared_x must be unhashed, 32-byte encoded x coordinate of the shared point
    • Use HKDF-extract with sha256, IKM=shared_x and salt=utf8_encode('nip44-v2')
    • HKDF output will be a conversation_key between two users.
    • It is always the same, when key roles are swapped: conv(a, B) == conv(b, A)
  2. Generate a random 32-byte nonce
    • Always use CSPRNG
    • Don't generate a nonce from message content
    • Don't re-use the same nonce between messages: doing so would make them decryptable, but won't leak the long-term key
  3. Calculate message keys
    • The keys are generated from conversation_key and nonce. Validate that both are 32 bytes long
    • Use HKDF-expand, with sha256, PRK=conversation_key, info=nonce and L=76
    • Slice 76-byte HKDF output into: chacha_key (bytes 0..32), chacha_nonce (bytes 32..44), hmac_key (bytes 44..76)
  4. Add padding
    • Content must be encoded from UTF-8 into byte array
    • Validate plaintext length. Minimum is 1 byte, maximum is 65535 bytes
    • Padding format is: [plaintext_length: u16][plaintext][zero_bytes]
    • Padding algorithm is related to powers-of-two, with min padded msg size of 32 bytes
    • Plaintext length is encoded in big-endian as first 2 bytes of the padded blob
  5. Encrypt padded content
    • Use ChaCha20, with key and nonce from step 3
  6. Calculate MAC (message authentication code)
    • AAD (additional authenticated data) is used - instead of calculating MAC on ciphertext, it's calculated over a concatenation of nonce and ciphertext
    • Validate that AAD (nonce) is 32 bytes
  7. Base64-encode (with padding) params using concat(version, nonce, ciphertext, mac)

Encrypted payloads MUST be included in an event's payload, hashed, and signed as defined in NIP 01, using schnorr signature scheme over secp256k1.

Decryption

Before decryption, the event's pubkey and signature MUST be validated as defined in NIP 01. The public key MUST be a valid non-zero secp256k1 curve point, and the signature must be valid secp256k1 schnorr signature. For exact validation rules, refer to BIP-340.

  1. Check if first payload's character is #
    • # is an optional future-proof flag that means non-base64 encoding is used
    • The # is not present in base64 alphabet, but, instead of throwing base64 is invalid, implementations MUST indicate that the encryption version is not yet supported
  2. Decode base64
    • Base64 is decoded into version, nonce, ciphertext, mac
    • If the version is unknown, implementations must indicate that the encryption version is not supported
    • Validate length of base64 message to prevent DoS on base64 decoder: it can be in range from 132 to 87472 chars
    • Validate length of decoded message to verify output of the decoder: it can be in range from 99 to 65603 bytes
  3. Calculate conversation key
  4. Calculate message keys
  5. Calculate MAC (message authentication code) with AAD and compare
    • Stop and throw an error if MAC doesn't match the decoded one from step 2
    • Use constant-time comparison algorithm
  6. Decrypt ciphertext
    • Use ChaCha20 with key and nonce from step 3
  7. Remove padding
    • Read the first two BE bytes of plaintext that correspond to plaintext length
    • Verify that the length of sliced plaintext matches the value of the two BE bytes
    • Verify that calculated padding from step 3 of the encryption process matches the actual padding

Details

Implementation pseudocode

The following is a collection of python-like pseudocode functions which implement the above primitives, intended to guide implementers. A collection of implementations in different languages is available at https://github.com/paulmillr/nip44.

# Calculates length of the padded byte array.

def calc_padded_len(unpadded_len):

  next_power = 1 << (floor(log2(unpadded_len - 1))) + 1

  if next_power <= 256:

    chunk = 32

  else:

    chunk = next_power / 8

  if unpadded_len <= 32:

    return 32

  else:

    return chunk * (floor((len - 1) / chunk) + 1)



# Converts unpadded plaintext to padded bytearray

def pad(plaintext):

  unpadded = utf8_encode(plaintext)

  unpadded_len = len(plaintext)

  if (unpadded_len < c.min_plaintext_size or

      unpadded_len > c.max_plaintext_size): raise Exception('invalid plaintext length')

  prefix = write_u16_be(unpadded_len)

  suffix = zeros(calc_padded_len(unpadded_len) - unpadded_len)

  return concat(prefix, unpadded, suffix)



# Converts padded bytearray to unpadded plaintext

def unpad(padded):

  unpadded_len = read_uint16_be(padded[0:2])

  unpadded = padded[2:2+unpadded_len]

  if (unpadded_len == 0 or

      len(unpadded) != unpadded_len or

      len(padded) != 2 + calc_padded_len(unpadded_len)): raise Exception('invalid padding')

  return utf8_decode(unpadded)



# metadata: always 65b (version: 1b, nonce: 32b, max: 32b)

# plaintext: 1b to 0xffff

# padded plaintext: 32b to 0xffff

# ciphertext: 32b+2 to 0xffff+2

# raw payload: 99 (65+32+2) to 65603 (65+0xffff+2)

# compressed payload (base64): 132b to 87472b

def decode_payload(payload):

  plen = len(payload)

  if plen == 0 or payload[0] == '#': raise Exception('unknown version')

  if plen < 132 or plen > 87472: raise Exception('invalid payload size')

  data = base64_decode(payload)

  dlen = len(d)

  if dlen < 99 or dlen > 65603: raise Exception('invalid data size');

  vers = data[0]

  if vers != 2: raise Exception('unknown version ' + vers)

  nonce = data[1:33]

  ciphertext = data[33:dlen - 32]

  mac = data[dlen - 32:dlen]

  return (nonce, ciphertext, mac)



def hmac_aad(key, message, aad):

  if len(aad) != 32: raise Exception('AAD associated data must be 32 bytes');

  return hmac(sha256, key, concat(aad, message));



# Calculates long-term key between users A and B: `get_key(Apriv, Bpub) == get_key(Bpriv, Apub)`

def get_conversation_key(private_key_a, public_key_b):

  shared_x = secp256k1_ecdh(private_key_a, public_key_b)

  return hkdf_extract(IKM=shared_x, salt=utf8_encode('nip44-v2'))



# Calculates unique per-message key

def get_message_keys(conversation_key, nonce):

  if len(conversation_key) != 32: raise Exception('invalid conversation_key length')

  if len(nonce) != 32: raise Exception('invalid nonce length')

  keys = hkdf_expand(OKM=conversation_key, info=nonce, L=76)

  chacha_key = keys[0:32]

  chacha_nonce = keys[32:44]

  hmac_key = keys[44:76]

  return (chacha_key, chacha_nonce, hmac_key)



def encrypt(plaintext, conversation_key, nonce):

  (chacha_key, chacha_nonce, hmac_key) = get_message_keys(conversation_key, nonce)

  padded = pad(plaintext)

  ciphertext = chacha20(key=chacha_key, nonce=chacha_nonce, data=padded)

  mac = hmac_aad(key=hmac_key, message=ciphertext, aad=nonce)

  return base64_encode(concat(write_u8(2), nonce, ciphertext, mac))



def decrypt(payload, conversation_key):

  (nonce, ciphertext, mac) = decode_payload(payload)

  (chacha_key, chacha_nonce, hmac_key) = get_message_keys(conversation_key, nonce)

  calculated_mac = hmac_aad(key=hmac_key, message=ciphertext, aad=nonce)

  if not is_equal_ct(calculated_mac, mac): raise Exception('invalid MAC')

  padded_plaintext = chacha20(key=chacha_key, nonce=chacha_nonce, data=ciphertext)

  return unpad(padded_plaintext)



# Usage:

#   conversation_key = get_conversation_key(sender_privkey, recipient_pubkey)

#   nonce = secure_random_bytes(32)

#   payload = encrypt('hello world', conversation_key, nonce)

#   'hello world' == decrypt(payload, conversation_key)

Audit

The v2 of the standard was audited by Cure53 in December 2023. Check out audit-2023.12.pdf and auditor's website.

Tests and code

A collection of implementations in different languages is available at https://github.com/paulmillr/nip44.

We publish extensive test vectors. Instead of having it in the document directly, a sha256 checksum of vectors is provided:

269ed0f69e4c192512cc779e78c555090cebc7c785b609e338a62afc3ce25040  nip44.vectors.json

Example of a test vector from the file:

{

  "sec1": "0000000000000000000000000000000000000000000000000000000000000001",

  "sec2": "0000000000000000000000000000000000000000000000000000000000000002",

  "conversation_key": "c41c775356fd92eadc63ff5a0dc1da211b268cbea22316767095b2871ea1412d",

  "nonce": "0000000000000000000000000000000000000000000000000000000000000001",

  "plaintext": "a",

  "payload": "AgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABee0G5VSK0/9YypIObAtDKfYEAjD35uVkHyB0F4DwrcNaCXlCWZKaArsGrY6M9wnuTMxWfp1RTN9Xga8no+kF5Vsb"

}

The file also contains intermediate values. A quick guidance with regards to its usage: