Penglai Enclave SDK Implementation

Penglai-Enclave's SDK includes two components: Enclave driver and enclave application SDK.

Enclave Driver

Currently, the enclave driver is for Linux host.

The implementation:

• penglai-enclave-driver
• penglai-elfloader
• penglai-ioctl
• penglai-page
• penglai-enclave

Penglai-enclave-driver.c

Driver init

During init, the driver will register a misc device, so that untrusted applications can operate with SM for enclave creation and management. ret=misc_register(&enclave_dev);

After that, the driver will allocate a range of memory, through __get_free_pages(GFP_HIGHUSER, DEFAULT_SECURE_PAGES_ORDER);

These memory will be passed to the SM for enclaves to use.

Device OPs

static const struct file_operations enclave_ops = {
  .owner = THIS_MODULE,
  .mmap = enclave_mmap,
  .unlocked_ioctl = penglai_enclave_ioctl
};

The above is the OPs provided by the device. Here, we allow mmap and ioctl to issue requests to enclave-driver. (notes: mmap is still a placeholder in open-sourced version)

Penglai-enclave-ioctl.c

This file implements the allowed enclave requests from (untrusted) user-space.

 case PENGLAI_ENCLAVE_IOC_CREATE_ENCLAVE:
     ret = penglai_enclave_create(filep, (unsigned long)ioctl_data);
      break;
    case PENGLAI_ENCLAVE_IOC_RUN_ENCLAVE:
      ret = penglai_enclave_run(filep, (unsigned long)ioctl_data);
      break;
    case PENGLAI_ENCLAVE_IOC_ATTEST_ENCLAVE:
      ret = penglai_enclave_attest(filep, (unsigned long)ioctl_data);
      break;
    case PENGLAI_ENCLAVE_IOC_STOP_ENCLAVE:
      ret = penglai_enclave_stop(filep, (unsigned long)ioctl_data);
      break;
    case PENGLAI_ENCLAVE_IOC_RESUME_ENCLAVE:
      ret = penglai_enclave_resume(filep, (unsigned long)ioctl_data);
      break;
    case PENGLAI_ENCLAVE_IOC_DESTROY_ENCLAVE:
      ret = penglai_enclave_destroy(filep, (unsigned long)ioctl_data);
      break;
    case PENGLAI_ENCLAVE_IOC_DEBUG_PRINT:
      ret = SBI_CALL_1(SBI_SM_DEBUG_PRINT, 0);
      break;

penglai_enclave_ioctl This is the entry of ioctl function. Currently we allow 1024 Bytes argument from user-space, which are sufficient to handle the CMDs penglai supported.

penglai_enclave_create

The function is for creating an enclave instance.

The user will pass a structure, penglai_enclave_user_param, to pass the parameters of an enclave to driver.

struct penglai_enclave_user_param
{
  unsigned long eid;
  unsigned long elf_ptr;
  long elf_size;
  long stack_size;
  unsigned long untrusted_mem_ptr;
  long untrusted_mem_size;
};

Here, the eid is a global ID for enclaves.

After the creation is finished, the driver will assign an unique eid to the untrusted app, so it can operate the enclave.

 enclave_param->eid = enclave_idr_alloc(enclave);

Notes: A potential issue is that a malicious use will try another eid to operate other users' enclaves. To avoid the issue, the secure monitor will check the sptbr before perform real operations. The OS will also ensure the ptbr (page table) is correct.

In the function, it will first invoke enclave = create_enclave(total_pages); to create enclave by allocating related pages.

Afterthat, penglai_enclave_eapp_preprare is invoked to load an enclave elf file into memory.

Then, Penglai will allocate a set of untrusted memory, for communication between host and enclaves.

  untrusted_mem_size = 0x1 << (ilog2(untrusted_mem_size - 1) + 1);
  if((untrusted_mem_ptr == 0) && (untrusted_mem_size > 0))
  {
    alloc_untrusted_mem(untrusted_mem_size, &untrusted_mem_ptr, enclave);
  }

Last, we will pass set of free memory (for monitor to use) and everything about an enclave to secure monitor, to finish the creation.

  ret = SBI_CALL_1(SBI_SM_CREATE_ENCLAVE, __pa(&enclave_sbi_param));

The logic of creation is quite complex, including allocating memory resources and invoking SM interfaces.

To ensure the correctness (especially on multi-core environment), Penglai uses a lock , spin_lock(&enclave_create_lock);, to protect the process.

sbi_param

Penglai uses a struct, penglai_enclave_sbi_param for communication with secure monitor.

struct penglai_enclave_sbi_param
{
  unsigned int * eid_ptr;
  unsigned long paddr;
  unsigned long size;
  unsigned long entry_point;
  unsigned long untrusted_ptr;
  unsigned long untrusted_size;
  unsigned long free_mem;
  unsigned long *ecall_arg0;
  unsigned long *ecall_arg1;
  unsigned long *ecall_arg2;
  unsigned long *ecall_arg3;
};

To ensure no information leakage, all values in the structure should be explicitly set, using the function create_sbi_param.

int create_sbi_param(enclave_t* enclave, struct penglai_enclave_sbi_param * enclave_sbi_param,
    unsigned long paddr, unsigned long size, unsigned long entry_point,
    unsigned long untrusted_ptr, unsigned long untrusted_size, unsigned long free_mem)
{
  enclave_sbi_param -> eid_ptr = (unsigned int* )__pa(&enclave -> eid);
  enclave_sbi_param -> ecall_arg0 = (unsigned long* )__pa(&enclave -> ocall_func_id);
  enclave_sbi_param -> ecall_arg1 = (unsigned long* )__pa(&enclave -> ocall_arg0);
  enclave_sbi_param -> ecall_arg2 = (unsigned long* )__pa(&enclave -> ocall_arg1);
  enclave_sbi_param -> ecall_arg3 = (unsigned long* )__pa(&enclave -> ocall_syscall_num);
  enclave_sbi_param -> paddr = paddr;
  enclave_sbi_param -> size = size;
  enclave_sbi_param -> entry_point = entry_point;
  enclave_sbi_param -> untrusted_ptr = untrusted_ptr ;
  enclave_sbi_param -> untrusted_size = untrusted_size;
  enclave_sbi_param -> free_mem = free_mem;
  return 0;
}

penglai_enclave_destroy

This function is for destroying an enclave.

First, enclave-driver needs to acquire an id, eid, from user-space, and uses get_enclave_by_id to get the structure, enclave_t.

The destroy_enclave(enclave); is the real function for destroying enclaves.

Last, enclave_idr_remove is used to free the eid.

penglai_enclave_run

To run an enclave, user can invoke the run cmd to enclave-driver.

Here, the driver invokes a monitor call, SBI_CALL_1(SBI_SM_RUN_ENCLAVE, enclave->eid), to run the enclave.

penglai_enclave_attest

For the design of attestation in Penglai, Please ref [Attestation Design](Attestation Design).

In the function, it will request SM to perform the real attestation.

  ret = SBI_CALL_3(SBI_SM_ATTEST_ENCLAVE, enclave->eid, __pa(&(enclave_param->report)), enclave_param->nonce);

penglai_enclave_stop

TODO

penglai_enclave_resume

TODO

Penglai-enclave-elfloader.c

This file is for loading an enclave application.

Enclave_eapp_prepare

penglai_enclave_eapp_prepare is the function to prepare an eapp, the definition is penglai_enclave_eapp_preprare(enclave_mem_t* enclave_mem, void* __user elf_ptr, unsigned long size, vaddr_t * elf_entry_point, vaddr_t stack_ptr, int stack_size)

It will receive a set of arguments which describe a running environment for an enclave, including the memory, code location, entry, and stack.

In the function, it will first prepare the stack for the encalve using enclave_alloc_page.

And the invoke penglai_enclave_loadelf(enclave_mem, elf_ptr, size, elf_entry_point) to load an enclave elf.

Penglai_enclave_loadelf

This function will parse an ELF passed from user-space. It checks the ELF header first, and then load different sections.

Two special case of sections are: NOBITS section and program.

map_untrusted_mem

int map_untrusted_mem(enclave_mem_t* enclave_mem, vaddr_t vaddr, paddr_t paddr, unsigned long size);

This function will map an untrusted memory to an enclave.

The definition of an untrusted memory: a continuous physical memory region, which can be accessed by both untrusted host and (one specific) enclave.

The purpose of untrusted memory is for data sharing between untrusted applications and enclaves.

The function will use map_va2pa(enclave_mem, vaddr, paddr, ENCLAVE_UNTRUSTED_PAGE); to map the region.

Before invoking map_untrusted_mem, enclave-driver must ensure that the size, and the two start addresses are valid. Otherwise, a user may attack the kernel.

Penglai-enclave-page.c

This file provides memory operations for enclave memory and page tables.

Notes: current open-sourced version still relies on the host kernel to manage enclave-PTs. However, the secure monitor will check the correctness before switching to enclave mode.

Supported functions:

vaddr_t enclave_alloc_page(enclave_mem_t* enclave_mem, vaddr_t vaddr, unsigned long flags);
void enclave_mem_int(enclave_mem_t* enclave_mem, vaddr_t vaddr, int size, paddr_t paddr);
vaddr_t get_free_mem(struct list_head* free_mem);
int enclave_mem_destroy(enclave_mem_t * enclave_mem);
vaddr_t map_va2pa(enclave_mem_t* enclave_mem, vaddr_t vaddr, paddr_t paddr, unsigned long flags);

Penglai-enclave.c

This file provides enclave-related operations.

Specifically,

• destroy_enclave
• create_enclave
• get_enclave_by_id
• enclave_idr_remove
• enclave_idr_alloc

enclave_idr_alloc

This function will allocate a new id for an enclave.

Encalve-driver uses Linux kernel's IDR mechanism to allocate the IDR.

A simplified code is:

  ueid = idr_alloc(&idr_enclave, enclave, ENCLAVE_IDR_MIN, ENCLAVE_IDR_MAX, GFP_KERNEL);
  
  if (ueid < ENCLAVE_IDR_MIN || ueid >= ENCLAVE_IDR_MAX) {
    error
  }

enclave_idr_remove

This function remove an idr through enclave = idr_remove(&idr_enclave, ueid);.

get_enclave_by_id

We can find an enclave through a unique id.

  enclave = idr_find(&idr_enclave, ueid); 

create_enclave

TODO

destroy_enclave

Most of the destroy work is to free allocated resources.

  kfree(enclave_mem);
  kfree(untrusted_mem);
  kfree(enclave);