General Feedback about Avalanche

[vlomonaco]

This was originally posted on the #avalanche slack channel of the continualAI workspace from Ludovic Denoyer. Reporting this to continue the discussion.

Ludovic Denoyer:
Please consider that I really like the work you are doing, and just providing a feedback that is subjective and specific to my own needs.

1. I am missing simple abstract classes to understand the key concepts (so I have to look at the documentation, but I have difficulties to understand the link between what I found in the doc, and the code). I would be happy to see a 'Strategy' abstract class with 'train' and 'eval', a 'Scenario' class, a 'Benchmark' class etc... all together in a 'core.py' file for instance. It would not change the way avalanche is working, but would help me to understand the key principles. You are using complex classes everywhere (which is nice) that makes the code difficult to read when you just want to understand the basics. Basically, by reading the core.py file, I would like to see if the platform is fitting my needs.
2. You have developed a very nice metrics system with the callbacks, but in many cases, the need will be: I have a (maybe already implemented) model in mind, I want a) to use my model on avalanche benchmarks b) compare the performance of my model w.r.t already implemented strategies. If I have to reimplement my own models with the callbacks, it is too complicated/expensive. So I think that many researchers will only use the 'from scratch mode' (but the callbacks are very nice if avalanche is building a catalog of baselines, and will be used at the level of the baselines). In the 'from scratch mode', then, I have two problems: a) I am missing a input/output specifications of the different functions (is train for one dataset, one full stream ?) b) I don't know how I can reuse the metrics plugin (e.g accuracy) in that case. Basically, what I need is strategy.train(task)->training_dynamics_metrics and some_metrics.compute(strategy)->metrics. The first one is implemented through callbacks in avalanche, and I don't really find the second one.
3. Minor point: the way you define your tasks seems specific to classification while one may be interested by doing regression, or even Reinforcement learning. If you include one additional level such as: Experience->ClassificationExperience then it would open to using avalanche not only for classification.

Andrea Cossu:

Hi @ludovic Denoyer and thank you for the feedback. This is something we really need right now, so don't worry. It is perfectly ok if you find something too obscure and you report it to us 🙂 Since I'm in charge of the evaluation part, I can answer for what the metrics are concerned. Using your own model or CL strategy should not affect the way you use metrics. Metrics are managed by the EvaluationPlugin which is passed to the CL strategy in the evaluator parameter. The EvaluationPlugin takes as parameters all the metrics you want to monitor (EpochAccuracy, StreamLoss , whatever) and the loggers you want to use to print the metrics (InteractiveLogger, TensorboardLogger and so on). The EvaluationPlugin is already integrated with the strategy callbacks and automatically provide to the metrics the information they need to compute their value. You don't need to call the update or result method of each metric. The Plugin will take care of that for you. This is an example showing the use of EvaluationPlugin . Let me know if this helps you or if I missed your point.

Antonio Carta:

Hi Ludovic, thank you for your feedback. I think all of us agree with you on most of the points.

1. This feels like a documentation problem, and we are working to address it. The idea of having a single core.py is interesting and helps documenting our code (especially for people that want to extend it), we can certainly do it.
2. Here, I think there are two separate points:
   Metrics provide classes (e.g. Accuracy) which are standalone, and you can use them without avalanche by calling update, reset, result explicitly. They also provide plugins which directly attach to the training/evaluation loops.
   Strategies are doing two different things: implementing generic train/eval loops and implementing the callback system that links together metrics, logger, and strategy plugins that you can use to develop custom strategies.
   I feel most CL strategies can be implemented by just defining a couple of callbacks. However, right now it's quite difficult to do it because the documentation is lacking. My objective would be to provide a couple of generic and easy to extend strategies that can be easily customized.
   Of course implementing strategies from scratch is always an option because you can use metrics and benchmarks standalone. The disadvantage is that you lose the automatic callbacks, so you need to call plugins and metrics yourself. Finding the correct tradeoff between customization and ease of reuse is difficult, and strategy plugins are trying to solve this problem. Right now they seem to work for our internal use but they require a lot of internal knowledge about the strategy, which I'm trying to simplify.
3. This is a WIP. I'm personally interested in this (not necessarily RL, but being less dependent on CV/classification). Hopefully, we will have it soon.

Ludovic:

Hi, yes, concerning the callback, I think that a certain amount of users will not dive into the callback system. From my own experience (the rlstructures library), users want to use their own logger, and to rely on few and simple abstractions. So, maybe, one solution would be to provide as an example the way to compute various metrics in the 'from scratch mode' which is not clear right now

Basically, right now, what I am 'taking' from avalanche is benchmarks/datasets since you provide the very nice catalog of benchmarks, but I am doing the evaluation of my models by myself, and there is certainly space here to provide some tools

concerning point 3), what we have here is a different definition of task (at the high-level) which is : Task.loss (which can be a classification loss, but also a reward function for instance)

But our solution is certainly too general 🙂

Andrea Cossu:

You are right. I think we should remove mentions about from scratch mode for the metrics. This is because if you want to use metrics (e.g. Accuracy) from scratch, you basically have to place update and result call inside specific callbacks.
The recommended way to use metrics is to let Avalanche do all the work for you. Moreover, metrics and loggers are kind of independent. We expect the user to implement his own logger, but mostly with existing metrics. Implementing a new logger does not require to use the from scratch mode of the metrics. The loggers only specifies a way to display the metric results and do not interfere with their computation, which is orchestrated by the EvaluationPlugin. So, basically, on each callback the new logger receives the metric values already computed and simply decide how to print them.
I think we should write a guide on how to create your own logger and integrate it into the training and evaluation flows. I will open an issue about this. Thank you!

Ludovic:

Sorry, maybe you misunderstood. My point is: how can I use metrics without using any callback 🙂 ?

Andrea:

Let's take the accuracy metrics as example (this is valid for all the others). There is a general class Accuracy which allows you to keep a running average of accuracies. You call update on each <x,y> pair and the class updates the running accuracy. With result you can get the current accuracy stored in the class and reset brings the class to its initial state. You can use an instance of Accuracy in your code by manually calling its methods when needed. A possible use case would be if you already have your training loop. You can insert the appropriate method calls into your training loop. However, if you use avalanche training functionalities, it is much simpler to use plugins metrics like StepAccuracy or EpochAccuracy, since you don't have to call any method manually 🙂

Vicenzo Lomonaco:

Thank you so much @ludovic Denoyer for your awesome feedback, which is incredibly useful at this stage! 🙂 I find in it a lot of confirmations about thoughts we already discussed with the dev team, and we will try to prioritize them.
Personally, I think the callback system is a very useful approach if we improve the doc and the example to make it easier to understand, even for people that just need to port their already developed strategy in avalanche (most of it can be learned just by looking at the BaseStrategy class more in depth).
Especially if you implement your strategy as a collection of plugins, this means other people will be able to build on it and use it more, significantly augmenting the impact of your research work. Being able to do something like this it's worth a few more hours of work I believe:

replay = ReplayPlugin(mem_size)
ewc = EWCPlugin(ewc_lambda)
strategy = BaseStrategy(
    model, optimizer, 
    criterion, mem_size,
    plugins=[replay, ewc])