Searchable Encryption applied to Firebase database

Firebase is a powerful and popular mobile backend offered “as a service” that you can start with it in minutes; it offers data storage, user authentication mechanisms and many other features for your mobile app. Data is stored as JSON and synchronized in realtime to every connected client and it’s encrypted on the wire (using HTTPS), but it remains unencrypted at the server/cloud side which makes Firebase not suitable for apps that need to store sensitive information like apps in mhealth or fintech.

Using IQrypt, you can build apps that store sensitive information like patient’s data and, at the same time, keeping the benefits of the simplicity and scalability of Firebase without sacrifice any Firebase features like queries/search even the data is encrypted.

Let’s take a look how easily you can integrate IQrypt within Firebase Android client; first let’s setup the Firebase app which is 1 minute task: just login on your Firebase account and put the name of your new app and press ‘CREATE NEW APP’:

firebaseNewApp

Your database is ready now, you can access it via the URL provided (eq: https://iqrypt.firebaseio.com/). On production, you will need to add authentication mechanisms, but for our demo, we’ll let it publicly accessible.

Now let’s go to the client and let’s start with our data model, first Patient entity:

public class Patient
{
    // Patient object contains following properties
    public String ObjectId ;
    public String SSN ;
    public String PhotoThumbnail ;
    public String PhotoDetail ;
    public String FirstName ;
    public String LastName ;
    public String Gender ;
    public String Location ;
    public String Building ;
    public int Floor ;
    public Date VisitDate;
    public String Treatment ;
    public String Status ;
    public String Doctor ;
    public PatientDetails PatientDetailsData;
    public String Version ;
    public boolean IsVisited;

}

PatientDetails contains other additional data (the definition can be found on Github repository).
The goal is that an entire Patient object is encrypted with RND encryption scheme and we will extract 2 tags that we want to search by:

  • SSN (Social Security Number) will be encrypted with DET so we can make equality queries
  • VisitDate will be encrypted with OPE so we can make range queries

Let’s create a Repository class that will manage our data and initialize it:

 

public class FirebaseRepository {

    Firebase patients;
    IEncryptor encryptorRND;
    IEncryptor encryptorDET ;
    IEncryptor encryptorOPE ;
    public FirebaseRepository()
    {

        Firebase myFirebaseRef = new Firebase("https://iqrypt.firebaseio.com/");
        patients=myFirebaseRef.child("patients");
        //init IQrypt
        IQryptConfigurator.setEncryptionChiper(Cipher.AES256, "my_super_secret");
        encryptorRND = EncryptorFactory.getEncryptor(EncryptionType.RND);
        encryptorDET = EncryptorFactory.getEncryptor(EncryptionType.DET);
        encryptorOPE = EncryptorFactory.getEncryptor(EncryptionType.OPE);
    }
....
}

Above, we got a reference to Patients JSON root node and we have also set IQrypt cipher + password(encryption key material) and create encryptors instances for the encryption schemes we need.

Now let’s create the method that will insert data into the database:

 public String createPatient(Patient patient) {

        Firebase newPatientRef = patients.push();


        patient.ObjectId = newPatientRef.getKey();
        Map<String, String> docContent = new HashMap<String, String>();

        docContent.put("patient", encryptorRND.encrypt(patient));
        //tags to search by
        docContent.put("ssn", encryptorDET.encrypt(patient.SSN));
        docContent.put("visitdate", encryptorOPE.encrypt(patient.VisitDate));

        newPatientRef.setValue(docContent);

        Log.d("LOG_PATIENT", "Patient with id:" + patient.ObjectId + " created");
        return newPatientRef.getKey();
    }

So when we call

Firebase newPatientRef = patients.push();

Firebase will generate a Key(which ca be used further to uniquely identify a patient), then we encrypt entire Patient object with RND encryption scheme, we’ll extract two tags that we will want to search by, and we’ll encrypt them with DET and respectively with OPE.

If you will call this method by passing a full Patient object, you will see on Firebase data browser something like this:

firebaseServer

We will implement now the methods that execute queries, first getBySSN:

public void  getPatientBySSN(String SSN) {
        String SSNEncrypted = encryptorDET.encrypt(SSN);
        Query queryRef = patients.orderByChild("ssn").equalTo(SSNEncrypted);


        queryRef.addChildEventListener(new ChildEventListener() {
            @Override
            public void onChildAdded(DataSnapshot snapshot, String previousChild) {

                String patientEncrypted=snapshot.child("patient").getValue(String.class);
                Patient pDec= (Patient) encryptorRND.decrypt(patientEncrypted,Patient.class);
                Log.d("LOG_PATIENT_BY_SSN", "Patient with SSN:" + pDec.SSN +" loaded" );
            }
.....
          });

}

We encrypt first the SSN and then send its encrypted value to Firebase engine which executes the query without decrypting the SSN.

Let’s see now how we can execute a range query using OPE scheme:

public void getPatientsVisitedBetween(Date start, Date end) {
        String encryptedStart=encryptorOPE.encrypt(start);
        String encryptedEnd=encryptorOPE.encrypt(end);

        Query queryRef = patients.orderByChild("visitdate").startAt(encryptedStart).endAt(encryptedEnd);

        queryRef.addChildEventListener(new ChildEventListener() {
            @Override
            public void onChildAdded(DataSnapshot snapshot, String previousChild) {

                String patientEncrypted=snapshot.child("patient").getValue(String.class);
                Patient pDec= (Patient) encryptorRND.decrypt(patientEncrypted,Patient.class);
                Log.d("LOG_PATIENT_BTW_DATES", "Patient with VisitDate:" + pDec.VisitDate +" loaded" );

            }
          
           ........
           });
}

So we have encrypted first the ‘start’ and ‘end’ values and then pass that values to Firebase query engine. The Firebase service will respond in the callback by loading only the Patient objects which has the VisitDate between specified dates (without decrypting anything).

Full source code can be found on Github.

Encrypt and stay safe!

IQrypt Java (Android) SDK source code is published

A few weeks ago we announced that we’ll release all IQrypt SDKs as open source. Today we are happy to release the Java(Android) version on Github. There is also a full sample that integrates IQrypt with Couchbase Lite and a bunch of Unit Tests, so you can inspect and see how IQrypt works.

We would be really happy to get feedback from the community, so go ahead, grab the sources and send us an email with your opinion about it.

Encrypt and stay safe!

Client side encryption for Couchbase Mobile

In the previous post we showed how you can integrate IQrypt SDK with Azure Mobile Apps by building a Windows Universal App which encrypts the data at the client side and store it further in SQL Azure.

Today  we’ll show how you can integrate IQrypt SDK with Couchbase Mobile. Couchbase Mobile empowers you to build mobile apps that store data locally and keep it in sync with Couchbase Server in the cloud via Sync Gateway. Couchbase Mobile already offers local database encryption and that is good, but this is not enough, as we showed in one of our blog posts.

We will make a demo app called DoctorsAgenda, it is a Xamarin Forms app that can run on Android and iOS and it uses latest version of Couchbase Lite SDK for .NET. The application is supposed to be used by a Doctor who makes visits to his patients in a hospital. The doctor can see his schedule and also can see patients medical data (which is stored locally and synchronized with Couchbase Server in the cloud) and then mark the patient as ‘Visited’. We will encrypt patient data at the client side( on the device) before the data is stored locally or sent to the cloud. At the end, the app will look like this on Android:

doctorsAgendaAndroid doctorsAgendaAndroid1

And like this on iOS:

doctorsAgendaiOS doctorsAgendaiOs1

Backend setup

First, let’s prepare the backend, we will install Couchbase Server and Couchbase Sync Gateway. For this demo we’ll use a Windows Server VM on Microsoft Azure. The installation on Windows is straight forward: download Couchbase Server Comunity Edition version 4.0.0 and then just Next, Next, etc. After the installation, it will be opened a browser with http://localhost:8091/ where you have to continue the setup (I let everything default). After the setup is done, go to ‘Data Buckets’ and, on the right-up corner, press ‘Create New Data Bucket’ and put as bucket name:patients

couchbaseIQrypt11

Now the server is ready, we’ll need to install Couchbase Sync Gateway; download it from the same location and install everything with default settings.

After the Sync Gateway is installed we’ll need to configure and link it with Couchbase Server.

So go to C:\Program Files (x86)\Couchbase\ and edit serviceconfig.json and modify it like this:

 
{
   "interface":":4984",
   "adminInterface":":4985",
   "log":["REST"],
   "databases":{
      "patients":{
         "users": {"GUEST": {"disabled": false, "all_channels": ["*"], "admin_channels": ["*"]}},
         "server":"http://localhost:8091",
         "bucket":"patients",
         "sync":`function(doc) {channel(doc.channels);}`
      }
   }
}

after that, start the Sync Gateway with this config file as argument:

couchbaseSyncGateway_started

Now the Sync Gateway can be accessed from localhost, but you will still have to do some extra settings to make it available on internet:

  • Open port 4984 in Windows Firewall on the Azure VM itself
  • Add a End Point on Azure VM and put ‘CB_sync_gateway’ for port 4984 (as public port and as private port)

Now our Sync Gateway can be accessed from outside( just be aware that we did no added any authentication and authorization mechanism, for production setups, follow Couchbase documentation about it).

syncGatewayFromOutside

Client setup

Let’s start with our data model, first Patient entity:

 
public class Patient
    {
        
        public string ObjectId { get; set; }
        public string PhotoThumbnail { get; set; }
        public string PhotoDetail { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Gender { get; set; }
        public string Location { get; set; }
        public string Building { get; set; }
        public string SSN { get; set; }
        public int Floor { get; set; }
        private DateTime _agenda_time;
        public DateTime AgendaTime 
        {
            get
            {
                return _agenda_time;
            }
            set
            {
                _agenda_time = value;
                String dtString = _agenda_time.Hour.ToString("00") + ":" + _agenda_time.Minute.ToString("00");
                AgendaHourMinute = dtString;
            }
        }

        public string AgendaHourMinute { get; set; }
        public string Treatment { get; set; }
        public string Status { get; set; }
        public string Doctor { get; set; }
        public string Version { get; set; }
        public PatientDetails PatientDetailsData { get; set; }
.......
}

PatientDetails contains other additional data (the definition can be found on Github repository).
The goal is that an entire Patient object is encrypted with RND encryption scheme and we will extract 2 tags that we want to search by:

  • SSN (Social Security Number) will be encrypted with DET so we can make equality queries
  • AgendaTime will be encrypted with OPE so we can make range queries

Couchbase stores data as JSON documents. Every document has some predefined properties/fields like ‘Id’ and ‘Rev’ and you can add a dynamic number of other properties/fields. In our use case we will have 3 properties: Patient, SSN and AgendaTime, so our JSON doc(encrypted) will look like this:

 
{
"Id":"-iHqlStW3Ike1B23datn4WQ",
"Patient":"flLjX9PhwaHlMn50uyJVyYllJUpsrK7ejfMadfadfsdfk99e3fjcj",
"SSN":"SYs8rlpHsUEV555Yp2ltGS7Qkon7yJy54pXgnPuQmSs=",
"AgendaTime":"@@@2103079269433"
.....
}

So, basically, within Patient property will be stored the entire Patient object encrypted with RND, within SSN will be store encrypted SSN value of the patient and similar for AgendaTime.

Now we will create a repository interface; its implementors will manage Couchbase Lite documents.

 

public interface IPatientRepository
 {
 
     Task<bool> ExistsPatients();
     Task<Patient> Get(string key);
     Task<IList<Patient>> GetByAgenda(DateTime agendatime, int limit);
     Task<Patient> GetBySSN(string sSN);
     Task Store(Patient item);
     Task StoreBatch(IList<Patient> items);
     Task Delete(Patient item);
     Task SynchronizeWithCloud();
 }

Now let’s start implementing the methods of this interface. We would need to add reference to Nuget packages:

Then compile IQrypt-iOS and IQrypt-Android projects from src/IQrypt/ and use IQrypt.dll from its /bin folders as references within DoctorsAgenda.iOS and DoctorsAgenda.Droid projects.

Then we will implement our repo; first we’ll do some initialization:

using Couchbase.Lite;
using IQrypt;
.....

 class CouchbaseMobileRepository : IPatientRepository
    {
        Manager dbmanager;
        Database database;
        IEncryptor encryptorRND;
        IEncryptor encryptorDET;
        IEncryptor encryptorOPE;

        public CouchbaseMobileRepository()
        {
            //initialize couchbase stuff
            dbmanager = Manager.SharedInstance;
            database = dbmanager.GetDatabase("patients");

            var ssnView = database.GetView("patientsBySSN");
            ssnView.SetMap((doc, emit) => emit((string)doc["SSN"], null), "2");

            var agendaView = database.GetView("patientsByAgendaDate");
            agendaView.SetMap((doc, emit) => emit((string)doc["AgendaTime"], null), "2");


            //initialize IQrypt
            IQryptConfigurator.SetEncryptionChiper(Cipher.AES256, "mysuper_secret");
            encryptorRND = EncryptorFactory.GetEncryptor(EncryptionType.RND);
            encryptorDET = EncryptorFactory.GetEncryptor(EncryptionType.DET);
            encryptorOPE = EncryptorFactory.GetEncryptor(EncryptionType.OPE);
        }
........

We have created above a Couchbase Lite database called ‘patients’ and two views: pattientsBySSN and patientsByAgendaDate.  Couchbase Views allows applications to create and maintain secondary indexes, so we can further query by SSN and AgendaTime( we’ll see further how).

We have also set IQrypt cipher + password(encryption key material) and create encryptors instances for the encryption schemes we need.

The app will check if we have any patients stored and if not, it will generate some patients with random info and then it will call StoreBatch method. So let’s implement this method:

 

public async Task StoreBatch(IList<Patient> patients)
        {
            foreach (var patient in patients)
            {
                var document = database.CreateDocument();

                var properties = new Dictionary<string, object>();
                properties["patient"] = encryptorRND.Encrypt(patient);
                properties["SSN"] = encryptorDET.Encrypt(patient.SSN);
                properties["AgendaTime"] = encryptorOPE.Encrypt(patient.AgendaTime);

                var revision = document.PutProperties(properties);

                patient.ObjectId = document.Id;
            }
        }

So, the data is encrypted with respective encryption schemes before the document is stored, so if it will further Sync with Couchbase Server, it will remain also encrypted.

We will implement now the methods that execute queries, first GetBySSN:

 public async Task<Patient> GetBySSN(string sSN)
        {
            Query query = database.GetView("patientsBySSN").CreateQuery();
            string SSNEncrypted = encryptorDET.Encrypt(sSN);

            query.Keys = new List() { SSNEncrypted };
            query.Limit = 1;
            var result = await query.RunAsync();
            var row = result.ToList().FirstOrDefault();
            if (row != null)
            {
                var documentProperties = row.Document.Properties;

                string patientEncrypted = documentProperties["patient"].ToString();
                Patient p = encryptorRND.Decrypt(patientEncrypted, typeof(Patient)) as Patient;

                p.ObjectId = row.DocumentId;
                return p;
            }
            return null;
        }

We encrypt first SSN and send its encrypted value to Couchbase engine which executes the query without decrypting the SSN.

Let’s see now how we can execute a range query using OPE scheme:

public async Task<IList<Patient>> GetByAgenda(DateTime agendatime, int limit)
        {
            Query query = database.GetView("patientsByAgendaDate").CreateQuery();
            string start = encryptorOPE.Encrypt(agendatime.Date);
            string end = encryptorOPE.Encrypt(agendatime.AddDays(1).Date);

            query.StartKey = start;
            query.EndKey = end;
            query.Limit = limit;

            var result = await query.RunAsync();
            var rows = result.ToList();
            List list = new List();
            foreach (var row in rows)
            {
                var documentProperties = row.Document.Properties;

                string patientEncrypted = documentProperties["patient"].ToString();
                Patient p = encryptorRND.Decrypt(patientEncrypted, typeof(Patient)) as Patient;

                p.ObjectId = row.DocumentId;
                list.Add(p);

            }
            return list;
        }

So we have encrypted first the ‘start’ and ‘end’ values and then set as StartKey and EndKey, respectively. The Couchbase engine will load only the documents within the specified range (without decrypting anything).

The data is stored locally so far. We will need to Sync with our backend, so let’s implement the method for Sync:

 public async Task SynchronizeWithCloud()
        {
            Replication pull = database.CreatePullReplication(CreateSyncUri());
            Replication push = database.CreatePushReplication(CreateSyncUri());
           
            pull.Start();
            push.Start();
            
        }
        Uri CreateSyncUri()
        {
            Uri syncUri = null;
            string scheme = "http";
            string host = "yourserver.cloudapp.net";
            int port = 4984;
            string dbName = "patients";

            var uriBuilder = new UriBuilder(scheme, host, port, dbName);
            syncUri = uriBuilder.Uri;

            return syncUri;
        }

After we’ll call this method, Replication process will start and will pull and push changes from/to Couchbase Server.
After the initial Sync(we generate some patients if db is empty), we can go on Couchbase Server and see how the data is stored:

couchbaseIQrypt1

And editing a Document:

couchbaseIQrypt2

So, as visible, everything remain encrypted, though the documents remains “browseable”.

Full source code can be found on Github.

Encrypt and stay safe!

Client side encryption for Azure Mobile Apps

IQrypt already integrates easily with Parse and Siaqodb via IQrypt plugins for Parse .NET SDK and Siaqodb and  today we’ll show how you can integrate IQrypt with Azure Mobile Apps and store data encrypted in Azure SQL Database. SQL Server 2016 and Azure SQL already offers Always Encrypted feature, however this is integrated only in ADO.NET driver  and is not available on Azure Mobile Client SDK which access the data via RESTful API.

Besides that, “Always Encrypted” offers only equivalent of DET(deterministic) and RND(randomized) encryption schemes, so you can only make equality/non equality queries using DET scheme.

IQrypt SDK offers many more schemes: RND(randomized), DET(deterministic), HMAC(keyed-hash), OPE(deterministic order preserving encryption), ROPE(randomized order preserving encryption) and BITMAP(client side bitmap index), empowering the server to execute many query types over encrypted data without decrypting it. A detailed description of those schemes that IQrypt SDK offers can be found here.

We will make a demo app called DoctorsRounds, it is a Windows Universal App supposed to be used by Doctors on a tablet or phone running Windows 10. Doctors will be able to visit patients in a hospital and fill information about those patients. We will encrypt patient data at the client side( on the device) before the data is sent to the cloud. At the end, the app will look like this:

doctorsrounds

So let’s start with the preparation, first we’ll need to setup our backend:

Creating a New Azure Mobile App

Inside of the Azure portal, simply select New -> Web + Mobile -> Mobile App, which is the starting point to configure your Azure Mobile App backend.

newapp

When selecting Mobile App in Azure, you will need to configure the service name (this is the URL where your backend  will live), configure your subscription, and set your resource group and plan.

appname

Add Data Connection

We can’t have a backend for our mobile apps without a database. Under the Data Connections section, select Add, and then configure a new SQL database , example bellow:

dataconnection

servercreation

Make sure that you keep Allow azure services to access server checked for your mobile app to allow services to properly connect to the database server. Also, be sure to keep your password in a secure place as you may need it in the future.

Click OK on all open windows and Azure will begin to create the database for us on the fly.

Adding ‘patients’ table

Under Mobile settings, look up for Easy Tables section, which enable us to easily set up and control the data coming to and from the client app. Select the Easy Tables section, and we’ll be prompted with a big blue warning asking us to configure Easy Tables/Easy APIs. Since we already setup the database, the only thing left to do is Initialize the app.

easytables1

 

After a few moments, the app will be fully initialized so we can add our first table of the database named patients. Easy Tables will automatically update and add the columns in the table dynamically based on the data we pass in. For this example, I’ll simply allow full anonymous access to the table, however it is possible to add authentication with Facebook, Twitter, Google, Microsoft, and other OAuth login providers.

easytables2

 

Our backend is fully functional, so let’s go to client now.

Client setup

We will define first the ‘Patient’ entity:

 

 

 
public class Patient
    {
        
        public string ObjectId { get; set; }
        public string PhotoThumbnail { get; set; }
        public string PhotoDetail { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Gender { get; set; }
        public string Location { get; set; }
        public string Building { get; set; }
        public string SSN { get; set; }
        public int Floor { get; set; }
        private DateTime _agenda_time;
        public DateTime AgendaTime 
        {
            get
            {
                return _agenda_time;
            }
            set
            {
                _agenda_time = value;
                String dtString = _agenda_time.Hour.ToString("00") + ":" + _agenda_time.Minute.ToString("00");
                AgendaHourMinute = dtString;
            }
        }

        public string AgendaHourMinute { get; set; }
        public string Treatment { get; set; }
        public string Status { get; set; }
        public string Doctor { get; set; }
        public string Version { get; set; }
        public PatientDetails PatientDetailsData { get; set; }
.......
}

PatientDetails contains other additional data (the definition can be found on Github repository).
The goal is that an entire Patient object is encrypted with RND encryption scheme and we will extract 3 tags that we want to search by:

  • SSN (Social Security Number) will be encrypted with DET so we can make equality queries
  • AgendaTime will be encrypted with OPE so we can make range queries
  • Building will be let as plain text since does not reveal sensitive info about the patient

To accomplish that, we will define the entity which will be stored in Azure SQL database as follow:

 

 

 
 [Microsoft.WindowsAzure.MobileServices.DataTable("patients")]
    public class AzurePatient
    {
        public string Id { get; set; }
        public string Patient { get; set; }
        public string SSN { get; set; }
        public string AgendaTime { get; set; }
        public string Building { get; set; }
    }

So, basically, within Patient property will be stored the entire Patient object encrypted with RND, within SSN will be store encrypted SSN value of the patient, similar for AgendaTime and Building properties.

Now we will create a repository interface; its implementors will manage AzurePatient objects and will be responsible with the communication with our Azure Mobile App backend:

 

 public interface IPatientRepository:IDisposable
 {
     Task InitRepo();
     Task<IList<Patient>> GetAll();
     Task<Patient> Get(string key);
     Task<IList<Patient>> GetByBuilding(string building, int limit);
     Task<IList<Patient>> GetByAgenda(DateTime agendatime, int limit);
     Task<Patient> GetBySSN(string sSN);
     Task Store(Patient item);
     Task StoreBatch(IList<Patient> items);
     Task Delete(Patient item); 
 
 }

Now let’s start implementing the methods of this interface. We would need to add references to Nuget packages:

Then compile IQrypt-UWP project from src/IQrypt/IQrypt-UWP and use IQrypt.dll from its /bin folder.

Then we will implement our repo, first we’ll do some initialization:

...
using Microsoft.WindowsAzure.MobileServices;
using IQrypt;

public class AzureMobileRepository : IPatientRepository
 {
    private MobileServiceClient mobileService;
    IEncryptor encryptorRND;
    IEncryptor encryptorDET;
    IEncryptor encryptorOPE;
    IMobileServiceTable<AzurePatient> table;
   public async Task InitRepo()
   {
     mobileService = new MobileServiceClient("https://iqryptdemo.azurewebsites.net");
     table = mobileService.GetTable<AzurePatient>();

     IQryptConfigurator.SetEncryptionChiper(Cipher.AES256, "mysuper_secret");
     encryptorRND = EncryptorFactory.GetEncryptor(EncryptionType.RND);
     encryptorDET = EncryptorFactory.GetEncryptor(EncryptionType.DET);
     encryptorOPE = EncryptorFactory.GetEncryptor(EncryptionType.OPE);
   }
.......

We have created above a reference to Azure ‘patients’ table, set IQrypt cipher + password(encryption key material) and create encryptors instances for the encryption schemes we need.
The app will check if we have any patients stored and if not, it will generate some patients with random info and the it will call StoreBatch method. So let’s implement this method:

 
 public async Task StoreBatch(IList<Patient> items)
        {
            foreach (Patient patient in items)
            {
                AzurePatient ap = new AzurePatient();

                ap.Patient = encryptorRND.Encrypt(patient);
                ap.AgendaTime = encryptorOPE.Encrypt(patient.AgendaTime);
                ap.SSN =encryptorDET.Encrypt( patient.SSN);
                ap.Building = patient.Building;
              
                ap.Id = Guid.NewGuid().ToString();
               
                await table.InsertAsync(ap);
            }
        }

So all data is encrypted with the respective encryption schemes before the object is sent to the cloud.

After calling this method, we can browse that data and we can see that the data is encrypted:

browsedata

Now let’s see how we will get the data back:

 
 public async Task<IList<Patient>> GetAll()
        {
          
            var allPatients=await table.ReadAsync();
            List<Patient> list = new List<Patient>();
            foreach (var azurePatient in allPatients)
            {
                Patient p = encryptorRND.Decrypt(azurePatient.Patient, typeof(Patient)) as Patient;
                p.ObjectId = azurePatient.Id;
                list.Add(p);
            }
            return list;
        }

So the only extra thing we had to do was to decrypt the Patient objects.
We will implement now the methods that executes queries, first GetBySSN:

 
 public async Task<Patient> GetBySSN(string SSN)
        {
            string ssnEncrypted = encryptorDET.Encrypt(SSN);
            var patients = await table.Where(p => p.SSN == ssnEncrypted).ToListAsync();
            if (patients.Count == 1)
            {
                Patient p = encryptorRND.Decrypt(patients[0].Patient, typeof(Patient)) as Patient;
                p.ObjectId = patients[0].Id;
                return p;
            }
            return null;
        }

First thing, we have encrypted the SSN we want to search by, then we pass the encrypted value to the server/cloud service and the server will execute the query and return the matched Patient object without decrypting anything.

Let’s see now how we can execute a range queries using OPE scheme:

  public async Task<IList<Patient>> GetByAgenda(DateTime agendatime, int limit)
        {
          
            string start =encryptorOPE.Encrypt(agendatime.Date);
            string end = encryptorOPE.Encrypt(agendatime.AddDays(1).Date);
            //load all patients with agenda time between start and end by OData query
            var allPatients = await table.ReadAsync("$filter=agendatime ge '" + start + "' and agendatime lt '" + end + "'&$top=" + limit);
            List<Patient> list = new List<Patient>();
            foreach (var azurePatient in allPatients)
            {
                Patient p = encryptorRND.Decrypt(azurePatient.Patient, typeof(Patient)) as Patient;
                p.ObjectId = azurePatient.Id;
                list.Add(p);
            }
            return list;
        }

As noticed, we run directly OData query because we cannot make LINQ query since the ‘start’ and ‘end’ are strings, but the server can execute queries using the operators ‘>=’  and ‘< ‘ even on string types.

Full source code of the app can be found on Github.  Take a look on the video bellow to see how to use the app:

 

Encrypt and stay safe!

Client side encryption performance – Raspberry Pi

IQrypt is designed to be used at client side, more specifically directly on client mobile/desktop devices, so every encryption/decryption operation is executed by the device itself. Performance is a critical aspect of any application and we are asked often: “what about performance?”. We designed IQrypt with performance in mind and we did a lot of testing regarding performance at the very early stage of the product to be sure the product is practical and it can be embedded into any application even it runs on devices with modest specs.

Today we are showing you some performance tests results made on a Rasberry Pi 2 Model B which has 900MHz quad-core ARM CPU and 1 GB of RAM and which looks like this:

Pi2ModB1GB_-comp

As operating system, we used latest Windows 10 IoT Core (version 10.0.14295.1000) and latest IQrypt SDK for .NET that can be found on Github.

We will encrypt/decrypt 1000 and respectively 10000 objects and we will measure the time elapsed to execute those operations. IQrypt SDK for .NET, before encryption, it serialize objects to JSON and after decryption it deserialize back to objects so we will measure the time elapsed first for serialization and then for serialization + encryption to be able to extract the time elapsed for encryption only (similar for decryption and deserialization).

We will follow our best-practice model, so let’s define an entity type first:

    public class Patient
    {

        public string ObjectId { get; set; }
        public string SSN { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Gender { get; set; }
        public string Location { get; set; }
        public string Building { get; set; }
        public int Floor { get; set; }
        public DateTime AgendaTime { get; set; }
        public string Treatment { get; set; }
        public string Status { get; set; }
        public string Doctor { get; set; }
        public string Version { get; set; }
        public bool IsVisited { get; set; }
       
    }

 

Then the Document entity which has Content + Tags:

        public class Document
        {
            public string Content;
            public Dictionary<string, string> Tags;
        }

Now let’s create a method to generate a Patient entity :

        private Random rnd = new Random();
        private Patient GeneratePatient()
        {
            Patient p = new Patient();
            p.ObjectId = rnd.Next().ToString();
            p.FirstName = "John"+rnd.Next();
            p.LastName = "Smith" + rnd.Next();
            p.SSN = "721-07-" + rnd.Next(1111, 9999);
            p.AgendaTime = DateTime.Now.Date;
            p.Doctor = "Dr. Joe Peterson";
            p.Treatment = "Lorem ipsum....";
            p.Status = "stable";
            p.Building = "A";
            p.Floor = 1;
            return p;
            
        }

We will encrypt entire Patient object with RND encryption scheme and we’ll extract two tags: SSN (social security number) and AgendaTime.

We will create 2 methods, one that creates and serializes/deserialize 1000/10000 documents and one that serialize/deserialize + encrypt/decrypt those objects.

        private void SerializeOnly()
        {

            var stopwatch = new Stopwatch();
            listBox.Items.Clear();
            listBox.Items.Add("Serialize documents started...");
            //warm-up
            Patient pw = this.GeneratePatient();
            string serializedPatientw= JsonConvert.SerializeObject(pw);
            string serializedSSNTagw = JsonConvert.SerializeObject(pw.SSN);
            string serializedAgendaTimeTagw = JsonConvert.SerializeObject(pw.AgendaTime);

            stopwatch.Start();
            List documents = new List();
            for (int i = 0; i < 1000; i++)
            {
                Patient p = this.GeneratePatient();
                string serializedPatient = JsonConvert.SerializeObject(p);
                string serializedSSNTag = JsonConvert.SerializeObject(p.SSN);
                string serializedAgendaTimeTag = JsonConvert.SerializeObject(p.AgendaTime);
                Document doc = new Document();
                doc.Content = serializedPatient;
                doc.Tags = new Dictionary<string, string>();
                doc.Tags.Add("SSN", serializedSSNTag);
                doc.Tags.Add("AgendaTime", serializedAgendaTimeTag);
                documents.Add(doc);
            }
            stopwatch.Stop();
            listBox.Items.Add("Serializing documents took:" + stopwatch.Elapsed);

            listBox.Items.Add("Deserializing documents started...");
            stopwatch.Start();
            foreach (Document d in documents)
            {
                Patient p = (Patient)JsonConvert.DeserializeObject(d.Content, typeof(Patient));
                string ssnDeserialized= (string)JsonConvert.DeserializeObject(d.Tags["SSN"], typeof(string));
                DateTime agendaTimeDeserialized = (DateTime)JsonConvert.DeserializeObject(d.Tags["AgendaTime"], typeof(DateTime));
            }
            stopwatch.Stop();
            listBox.Items.Add("Deserializing documents took:" + stopwatch.Elapsed);
            
        }

And now also encrypt them (IQrypt serialize/deserialize objects internally):

        private void SerializeAndEncrypt()
        {
            listBox.Items.Clear();
           
            IQrypt.IQryptConfigurator.SetEncryptionChiper(IQrypt.Cipher.AES256, "mysuper_secret");
            var encryptorRND = IQrypt.EncryptorFactory.GetEncryptor(IQrypt.EncryptionType.RND);
            var encryptorDET = IQrypt.EncryptorFactory.GetEncryptor(IQrypt.EncryptionType.DET);
            var encryptorOPE = IQrypt.EncryptorFactory.GetEncryptor(IQrypt.EncryptionType.OPE);
          
            //warm-up 
            Patient pw = this.GeneratePatient();
            string encryptedPatientw = encryptorRND.Encrypt(pw);
            string encryptedSSNTagw = encryptorDET.Encrypt(pw.SSN);
            string encryptedAgendaTimeTagw = encryptorOPE.Encrypt(pw.AgendaTime);

            listBox.Items.Add("Encrypt documents started...");
            var stopwatch = new Stopwatch();
            stopwatch.Start();
            List documents = new List();
            for (int i = 0; i < 1000; i++)
            {
                Patient p = this.GeneratePatient();
                string encryptedPatient = encryptorRND.Encrypt(p);
                string encryptedSSNTag = encryptorDET.Encrypt(p.SSN);
                string encryptedAgendaTimeTag = encryptorOPE.Encrypt(p.AgendaTime);
                Document doc = new Document();
                doc.Content = encryptedPatient;
                doc.Tags = new Dictionary<string, string>();
                doc.Tags.Add("SSN", encryptedSSNTag);
                doc.Tags.Add("AgendaTime", encryptedAgendaTimeTag);
                documents.Add(doc);
            }
            stopwatch.Stop();
            listBox.Items.Add("Encrypting documents took:" + stopwatch.Elapsed);

            listBox.Items.Add("Decrypt documents started...");
            stopwatch.Start();
            foreach (Document d in documents)
            {
                Patient p = (Patient)encryptorRND.Decrypt(d.Content, typeof(Patient));
                string ssnDecrypted = (string)encryptorDET.Decrypt(d.Tags["SSN"], typeof(string));
                DateTime agendaTimeDecrypted = (DateTime)encryptorOPE.Decrypt(d.Tags["AgendaTime"], typeof(DateTime));
            }
            stopwatch.Stop();
            listBox.Items.Add("Decrypting documents took:" + stopwatch.Elapsed);

          
        }

 

The above methods are called in a Universal Windows App and executed on the Raspberry Pi device, here are the results:

Encrypting:

Nr.Objects Serialize(ms) Encrypt+Serialize(ms) Encryption(ms) Avg per object(ms)
1000 677 3289 2612 2.612
10000 6069 19040 12971 1.2971

Decrypting:

Nr.Objects Deserialize(ms) Decrypt+deserialize(ms) Decryption(ms) Avg per object(ms)
1000 2366 4643 2277 2.277
10000 15186 33099 17913 1.7913

So if we encrypt 1000 objects, it took 2612 milliseconds (col4= col3-col2) just for the encryption operation, so an average of 2.6 milliseconds per object. If we encrypt 10000 objects the average is even lower 1.29 milliseconds per object.
Similar for decryption, the average time for decrypting an object is 2.27 milliseconds when we decrypt 1000 objects and 1.79 ms when we decrypt 10000 objects.

Despite the fact that Raspberry Pi has modest specs comparing with many other mobile phones/devices nowadays, IQrypt overhead is between 1 and 3 milliseconds per object/document which is really ignorable comparing with network latency and other factors which influence the performance of an application.

Searchable Encryption, is it secure?

IQrypt uses two main types of encryption schemes:

  1. Probabilistic
  2. Deterministic

An accurate definition of probabilistic encryption can be found on Wikipedia, basically it means that if we encrypt a message with a certain key it will yield, in general, to a different cipher text at every call of the encryption function. Such scheme it is considered to be semantically secure .

In contrast, if we encrypt a message M with a deterministic encryption scheme with a key K it will always yield to the same cipher text C. Deterministic encryption schemes are vulnerable to statistical frequency analysis attacks.  Basically the attacker may learn information by checking the frequency of a certain cipher text in the database. However if deterministic schemes are used properly, it may offer the same level of security as probabilistic ones. Let’s analyze it more deeply.

As probabilistic schemes, IQrypt SDK provides :

  • RND – Randomized encryption
  • ROPE – Randomized order preserving encryption
  • BITMAP – Client side bitmap indexes

As deterministic encryption schemes, IQrypt SDK provides:

  • DET – Deterministic encryption
  • HMAC – Keyed-hash scheme
  • OPE – Order preserving encryption

Let’s see how IQrypt SDK uses those schemes to provide maximum security.

IQrypt works best with Document Oriented Databases like MongoDB or Couchbase, so the goal is to encrypt JSON documents. Usually JSON Documents are stored in Collections, a collection can be associated with Tables in relational databases. Every document consists in a list of key-value pairs, the values can be also another list of key-value pairs, arrays or simple types. Within every document it is embedded a special field called “Id” – unique identifier within a collection.

To obtain the desired security level, we will modify this data model, so our Document will have the following model:

  • Id – unique identifier
  • Content – document content as described above and where resides all data.
  • Tags – a simple key-value pair list

Our process flow will be as following: before encryption, we will extract the keywords from the Document content that we want to search by and put them as Tags. So in fact, we will store some data redundantly, once inside Document content and once within the Tags. The encryption will be done as follow:

  • Id – will be let plain text because usually should not reveal anything sensitive.
  • Content will be encrypted RND.
  • Each tag value will be encrypted with different encryption scheme based on the search needs:
    • HMAC – keyed-hash message authentication code used for equality/non-equality queries
    • DET – deterministic encryption used for equality /non-equality queries
    • OPE – deterministic order preserving encryption used for equality/non-equality queries and also for range queries
    • ROPE – randomized order preserving encryption used for equality/non-equality queries and also for range queries
    • BITMAP – client side encrypted bitmap indexes used for low cardinality tags capable to sustain equality/non-equality queries and also range queries
    • PLAINTEXT – plain text, no encryption, all types of queries

RND- Randomized encryption scheme

Document content is encrypted with this scheme.This scheme uses AES and/or Camellia as block ciphers with a key size of 128 and/or 256 bits and as mode of operation we use CBC with a random initial vector (IV). Such scheme is semantically-secure providing indistinguishability under chosen plaintext attack (IND-CPA).

HMAC – Keyed-hash scheme

HMAC – keyed-hash authentication it is one-way function, it means once you apply this function over a plain text you cannot reverse it. For this scheme we use SHA-256 as hashing algorithm. This construction allows us to make equality/non-equality queries for Tag values. Another important feature of this function is the fact that for any input size, the output has always the same size (256 bits in our case), so you can combine, for example, multiple fields values from the Document and put it as one Tag. Decryption of this tag value is not needed in our case, because we have the actual value within the document anyway, so using such a query, the server will return us the matched documents and we’ll get the values after we decrypt the document; the Tag being used only for the search purpose. Being a deterministic scheme, an adversary might perform a statistical analysis. To avoid that, we recommend to use this scheme when you want to search by multiple fields, example: FirstName, LastName and ZipCode and when those fields/tags combined has very high cardinality, ideally unique per collection. In this way it will provide the same security level as RND.

DET – Deterministic encryption scheme

Deterministic encryption scheme is used to encrypt Tag values and letting the cloud server to run equality or non-equality queries. For this scheme we use AES and/or Camellia block ciphers in CBC mode with synthetic IV (initial vector). Being a deterministic scheme, an adversary might perform a statistical analysis. To avoid that, we recommend to use this scheme when Tags values have high cardinality, ideally unique per collection. Example of such values: SSN(Social Security Number), Email, etc; such values are usually unique per collection. In this way it will provide the same security level as RND.

OPE – Order Preserving Encryption

This is a deterministic encryption scheme and it is used to encrypt Tag values and letting the cloud server to run equality or non-equality queries but also range queries. This scheme is very efficient because it let the server create indexes and make queries as efficiently as for unencrypted data. Besides the equality, the encrypted values leaks also the order which can be insecure if it is encrypted values with low cardinality. Therefor, we recommend to use this scheme to Tags values with high cardinality, example: DateTime and when not all (possible) values of plain text can be found as cipher text in the database. In this way it will provide a strong security level.

ROPE – Randomized Order Preserving encryption scheme

This is a probabilistic encryption scheme and it is used to encrypt Tag values and letting the cloud server to run equality or non-equality queries but also range queries. This scheme is very efficient because it let the server create indexes and make queries as efficiently as for unencrypted data. It is probabilistic because the cipher text is randomized, so for the same plain text and same encryption key, the cipher text will be different (with a certain probability) for consecutive calls of the algorithm. In this way there will be no frequency information leaked, however the order is still preserved, so if we have 2 messages a and b if a<b then Enc(a)<Enc(b). As noticed, the server cannot execute directly equality queries since Enc(a), most likely, is different on each run; however the equality queries are still possible by transforming the client equality query in a range query. From security perspective, this scheme is recommended to encrypt Tags with medium to high cardinality.

BITMAP – Client side bitmap indexes

Bitmap indexes work well for low-cardinality tags, which have a modest number of distinct values, either absolutely, or relative to the number of records/documents that contain the data. The extreme case of low cardinality is Boolean data (True, False). Bitmap indexes(compared with B-Tree structures) are very efficient about storage; for example if we have 1 million JSON documents, the total index size for holding a tag value is about 100KB. If we have a Boolean tag it will be needed to store 2 indexes, one for True values and one for False values so in total we will have 200KB. Usually bitmap indexes are kept in RAM; for a server, nowadays, this amount of data means nothing, but if we keep it on the client and if cardinality is higher, then storage may be problematic. Fortunately the bitmap indexes can be compressed and reduce drastically its size. IQrypt uses EWAH described here and thanks to it, we can use bitmap indexes only on the client side. So basically this type of index is fully encrypted on the client side with RND encryption scheme and stored encrypted on the server. When we run a query for a Tag value encrypted with this scheme, we will execute in fact 2 queries:one that download the entire (compressed and encrypted) bitmap index and one to get the documents that matches the query by providing to the server the identifiers got from bitmap indexes.( As an idea, we tested on many real world scenarios and the entire index compressed with EWAH for low cardinality tag values usually remain under 10KB). So we recommend to use this scheme if you have low cardinality tag values like Boolean, Sex, Statuses or even medium cardinality like for example Salary or Country if there is a decent total amount of documents stored in a collection. Since it is encrypted with RND it offers same degree of security as RND scheme.

To summarize: entire JSON document is always encrypted with most secure encryption scheme RND. The keywords (Tags) can be encrypted with different encryption scheme, BITMAP and ROPE for low to medium cardinality; DET, HMAC and OPE for high cardinality and PLAINTEXT for non-sensitive values.

Using combinations of these encryption schemes, IQrypt will allow you to store data highly secure and at the same time allow server to execute queries without decrypting the data.

The SDK is open source and it can be found on Github.

Encrypt and stay safe!

“We encrypt data in transit and at rest” cliche

We often see that cloud providers states that they “encrypt data in motion/in transit and also at rest”. Let’s see how this technically works.

Protection of Data-In-Motion

Data-in-motion is encrypted usually using TLS/SSL protocol, so the data is encrypted while is transferred from client to the cloud and is not anymore vulnerable to eavesdropping and tampering, but once it arrives on server, data is decrypted and processed by the database service and then stored.

Protection of Data-At-Rest

Some of cloud providers allow customers to encrypt data at rest – so called transparent encryption. Transparent Data Encryption is a technology that encrypts database files. TDE solves the problem of protecting data at rest, encrypting databases both on the hard drive and consequently on backup media. This technology was useful when you have your own datacenter but it is mostly useless in our new “cloud era” because data is exposed unencrypted to the database service.

This typical scenario is described in the picture bellow:
withoutIQrypt2

As visible, this is not enough; at the service level data is unencrypted so if a cyber criminal succeed to access the service, he will have access to all data, despite the fact that the data is encrypted or not “at rest”. The same principle applies for the cloud provider itself, he can still mine customer data for his own profit.

The alternative is client side encryption, so the data is encrypted before it leaves the client’s devices and stays encrypted on all levels. But client side encryption brings some challenges and the main important one is SEARCH. Typically, once the data is encrypted it cannot be searched anymore and the server cannot execute any computation over encrypted data. Another important challenge is the management of the encryption keys because that needs to be handled on the client device.

IQrypt aims to solve these challenges by providing searchable encryption algorithms and also a solution to manage the encryption keys securely called IQryptVault. The SDK is provided as open source and can be found on Github and IQryptVault will be provided also as open source very soon.

Encrypt and stay safe!

IQrypt is now open source

We are really happy to announce that based on feedback we got, we will Open Source all our SDKs. We started today by releasing the .NET SDK and on coming weeks, we’ll release also the Java(Android) SDK and also Swift(iOS) SDK.

IQrypt SDK is an encryption library that can be embedded into Mobile or IoT applications and its goal is to encrypt data that will be stored in a cloud database but preserving the ability to query that data without decrypting it.  This .NET version we are releasing today provides libraries for .NET 4.5, Universal Windows Platform, Xamarin.iOS and Xamarin.Android.

We eagerly waiting for feedback from community and strongly committed to change the way that companies store the data in the cloud.

You can inspect the source code on Github ; we will be really happy to get feedback from you!

Encrypt and stay safe!