Trying to read into array multiple times

[VwertIX]

So I'm reading json data from the NIST NVD products and vulnerabilities api and its got a limit on how many items it shows at once so you have to load the page multiple times to get the data, and I used glz::read to read the data from the api but when I read it the second time it just overwrites the array in my struct instead of emplacing back or pushing back the new data.

for reference my structs are like this

struct deprecatedBy {
    std::optional<string> cpeName;
    std::optional<string> cpeNameId;
};

struct deprecate {
    std::optional<string> cpeName;
    std::optional<string> cpeNameId;
};

/**
 * Internet resource for CPE
 */
struct defReference {
    string ref;
    std::optional<string> type;
};

/**
 * Human readable title for CPE
 */
struct defTitle {
    string lang;
    string title;
};

struct cpe {
    string cpeName;
    string cpeNameId;
    string created;
    bool deprecated;
    std::optional<std::vector<deprecatedBy>> deprecatedBy;
    std::optional<std::vector<deprecate>> deprecates;
    string lastModified;
    std::optional<std::vector<defReference>> refs;
    std::optional<std::vector<defTitle>> titles;
};

struct defCpe {
    cpe cpe;
};

struct cpeResult {
    string format;
    /**
     * NVD feed array of CPE
     */
    std::vector<defCpe> products;
    int64_t resultsPerPage;
    int64_t startIndex;
    string timestamp;
    int64_t totalResults;
    string version;
};

and this is a snippet of the code im using to read from the api

    cpr::Response r = cpr::Get(cpr::Url{ "https://services.nvd.nist.gov/rest/json/cpes/2.0" }, cpr::Timeout{ 30s });
    r.status_code;                  // 200
    r.header["content-type"];       // application/json; charset=utf-8
    r.text;                         // JSON text string

    glz::context ctx{};
    auto s = glz::read<glz::opts{ .error_on_unknown_keys = false }, cpeResult>(cpeResults, r.text, ctx);

Any idea what im doing wrong, i guess i could create a second cpeResult variable and then like push_back the data from it to the other one but theres probably a better way to do it.

Also I was wondering whats the difference between glz::read and glz::read_json.

Links to the nist api pages in case it helps.
https://nvd.nist.gov/developers/vulnerabilities
https://nvd.nist.gov/developers/products

[stephenberry]

glz::read<glz::opts{}> is the more flexible reading API that lets you set options and change the format. glz::read_json doesn't accept options and is specifically for JSON. If you're customizing options you'll use glz::read and glz::write. One tip is that you can use a constexpr variable for the options and pass it to your various read/write calls.

As for appending data to your vectors, this feature could be added as a compile time option/Glaze wrapper (to selectively choose which vectors to append to within glz::meta).

Currently you'd need to parse into separate instances and then combine them. I'll keep this issue alive until the feature has been added.

Thanks for the feedback!

[stephenberry]

@VwertIX, so I just merged in support (into main) for a new option .append_arrays, which will append to types like std::vector rather than overwriting.

Here are unit tests showing the support. The use in glz::meta is just in case you want to turn on the feature for only select vectors.

struct append_obj
{
   std::vector<std::string> names{};
   std::vector<std::array<int, 2>> arrays{};
};

template <>
struct glz::meta<append_obj>
{
   using T = append_obj;
   static constexpr auto value = object("names", append_arrays<&T::names>, "arrays", append_arrays<&T::arrays>);
};

suite append_arrays_tests = [] {
   "append_arrays vector"_test = [] {
      std::vector<int> v{};
      constexpr glz::opts append_opts{.append_arrays = true};
      expect(not glz::read<append_opts>(v, "[1,2,3]"));
      expect(v == std::vector<int>{1,2,3});
      expect(not glz::read<append_opts>(v, "[4,5,6]"));
      expect(v == std::vector<int>{1,2,3,4,5,6});
   };
   
   "append_arrays deque"_test = [] {
      std::deque<int> v{};
      constexpr glz::opts append_opts{.append_arrays = true};
      expect(not glz::read<append_opts>(v, "[1,2,3]"));
      expect(v == std::deque<int>{1,2,3});
      expect(not glz::read<append_opts>(v, "[4,5,6]"));
      expect(v == std::deque<int>{1,2,3,4,5,6});
   };
   
   "append_arrays append_obj"_test = [] {
      append_obj obj{};
      expect(not glz::read_json(obj, R"({"names":["Bob"],"arrays":[[0,0]]})"));
      expect(obj.names == std::vector<std::string>{"Bob"});
      expect(obj.arrays == std::vector<std::array<int, 2>>{{0,0}});
      
      expect(not glz::read_json(obj, R"({"names":["Liz"],"arrays":[[1,1]]})"));
      expect(obj.names == std::vector<std::string>{"Bob", "Liz"});
      expect(obj.arrays == std::vector<std::array<int, 2>>{{0,0},{1,1}});
   };
};