Migrate Complex Open DB Code

[中文版 (Chinese)]

1. background

In many scenarios, with RocksDB's legacy code, the Open DB code is very complex, and there may even be a framework for managing DB/ColumnFamily (e.g., flink's rocksdb state backend).

Migrating such a project to ToplingDB and using SidePluginRepo to configure and manage DB/ColumnFamily objects would require a lot of changes, so we designed a new scenario to reduce the cost of this code migration.

2. Inject the open DB into SidePluginRepo

This feature was initially supported in SidePluginRepo a long time ago, but at first it was considered to be a minor feature without much design consideration, and now we have refined and tested it.

2.1. Primary interface（ C++ ）

class SidePluginRepo { // Omit unrelated members

public:
  // The caller should ensure DB handle's life time is longer than SidePluginRepo
  void Put(const std::string& name, DB*);
  void Put(const std::string& name, DB*, const std::vector<ColumnFamilyHandle*>&);
  void Put(const std::string& name, DB_MultiCF*);

  void Put(const std::string& name, const std::shared_ptr<Options>&);
  void Put(const std::string& name, const std::shared_ptr<DBOptions>&);
  void Put(const std::string& name, const std::shared_ptr<ColumnFamilyOptions>&);

  bool Get(const std::string& name, std::shared_ptr<Options>*) const;
  bool Get(const std::string& name, std::shared_ptr<DBOptions>*) const;
  bool Get(const std::string& name, std::shared_ptr<ColumnFamilyOptions>*) const;

  Auto Get(const std::string& name) const; // C++ trick: shared_ptr<DBOptions> dbo = repo["dbo"];
  Auto operator[](const std::string& name) const; // C++ trick
};

2.2. Primary interface（ Java ）

class SidePluginRepo { // Omit unrelated members
    public void put(String name, RocksDB db);
    public void put(String name, String spec, Options opt);
    public void put(String name, String spec, DBOptions dbo);
    public void put(String name, String spec, ColumnFamilyOptions cfo);

    // will get a clone on each call, to sync, put after modified
    public ColumnFamilyOptions getCFOptions(String name);
    public DBOptions getDBOptions(String name);
}

3. how to use

3.1. C++ demo

// begin new code 1
SidePluginRepo repo;
repo.ImportAutoFile(json_or_yaml_file1); // basic conf
shared_ptr<DBOptions> dbo = repo["dbo"];
shared_ptr<ColumnFamilyOptions> cfo = repo["cfo"];
// end new code 1

OldCodeUpdateDBO(dbo);
OldCodeUpdateCFO(cfo);

// maybe update "dbo" and "cfo", "basic conf" and "update conf" can be
// both present or just one of the two.
// here dbo/cfo are from old user code, we put them to repo
repo.Put("dbo", dbo);
repo.Put("cfo", cfo);
repo.ImportAutoFile(json_or_yaml_file2); // update conf

// begin old code, open db, omit error check
vector<ColumnFamilyOptions> cf_desc;
vector<ColumnFamilyHandle*> cf_handles;
Add_Many_cfo_to(cf_desc);
DB* db = nullptr;
std::string dbpath = FromSomeConf(); // DB::Open's name param is really path
DB::Open(dbpath, cf_desc, &cf_handles, &db);
// end old code

// begin new code 2
std::string dbname; // name, not path, new concept of SidePluginRepo
repo.Put(dbname, db, cf_handles);
repo.StartHttpServer();
// end new code 2

// old code ...

repo.CloseAllDB(false); // new code 3, close

In general, with these changes, the migration of RocksDB to ToplingDB is complete.

In which, we can import option from json/yaml file first, modify option with code, and then import another json/yaml file to modify option at the end, and the two importautofiles can be reduced to one according to the specific situation. The general principle is: the back covers the front.

3.2. Java demo

In Java, importAutoFile is almost identical to C++. The main difference is that importautofile in Java must then be re-used to get the corresponding object reference in the repo:

repo.put("dbo", dbo);
repo.put("cfo", cfo);
repo.importAutoFile(json_or_yaml_file2); // update conf
dbo = repo.getDBOptions("dbo"); // update to dbo in java
cfo = repo.getCFOptions("cfo"); // update to cfo in java

Another difference:

// C++ Code:
// shared_ptr<DBOptions> dbo = repo["dbo"];
// shared_ptr<ColumnFamilyOptions> cfo = repo["cfo"];

// Java Code:
DBOptions dbo = repo.getDBOptions("dbo");
ColumnFamilyOptions cfo = repo.getCFOptions("cfo");

Full example:

// begin new code 1
SidePluginRepo repo = new SidePluginRepo();
repo.importAutoFile(json_or_yaml_file1); // basic conf
DBOptions dbo = repo.getDBOptions("dbo");
ColumnFamilyOptions cfo = repo.getCFOptions("cfo");
// end new code 1

oldCodeUpdateDBO(dbo);
oldCodeUpdateCFO(cfo);

// maybe update "dbo" and "cfo", "basic conf" and "update conf" can be
// both present or just one of the two.
// here dbo/cfo are from old user code, we put them to repo
repo.put("dbo", dbo);
repo.put("cfo", cfo);
repo.importAutoFile(json_or_yaml_file2); // update conf
dbo = repo.getDBOptions("dbo"); // update to dbo in java
cfo = repo.getCFOptions("cfo"); // update to cfo in java

// begin old code, open db, omit error check
List<ColumnFamilyOptions> cf_desc = new ArrayList<ColumnFamilyOptions>();
List<ColumnFamilyHandle> cf_handles = new ArrayList<ColumnFamilyHandle>();
// Add_Many_cfo_to(cf_desc); // weird, rocksdb jni cfname is byte[]
   cf_desc.add(new ColumnFamilyDescriptor("cfo".getBytes(), cfo));
   cf_desc.add(new ColumnFamilyDescriptor("more cfo".getBytes(), more_cfo));
String dbpath = FromSomeConf(); // DB::Open's name param is really path
RocksDB db = RocksDB.open(dbpath, cf_desc);
// end old code

// begin new code 2
String dbname; // name, not path, new concept of SidePluginRepo
repo.put(dbname, db, cf_handles);
repo.startHttpServer();
// end new code 2

// old code ...

repo.closeAllDB(); // new code 3, close

Java code, as usual, two importAutoFile, according to the specific circumstances, can be reduced to one, basically see in Java code coverage in the json/yaml configuration items with the same (DBOptions/ColumnFamilyOptions member), Again, use json/yaml to override the configuration item with the same name set in the Java code. The general principle is: the back covers the front.