From e01139a7c8d2bb1729a2453e7835a1b86eff77a2 Mon Sep 17 00:00:00 2001 From: Sanne Grinovero Date: Wed, 17 Jul 2024 16:17:35 +0100 Subject: [PATCH] Exploring Project Leyden --- _data/authors.yaml | 4 + _posts/2024-08-06-quarkus-and-leyden.adoc | 275 ++++++++++++++++++++++ assets/images/posts/leyden/AoT_vs_JiT.svg | 102 ++++++++ 3 files changed, 381 insertions(+) create mode 100644 _posts/2024-08-06-quarkus-and-leyden.adoc create mode 100644 assets/images/posts/leyden/AoT_vs_JiT.svg diff --git a/_data/authors.yaml b/_data/authors.yaml index 6d33b5de276..760efc5c503 100644 --- a/_data/authors.yaml +++ b/_data/authors.yaml @@ -19,6 +19,10 @@ sannegrinovero: job_title: "Hibernate team lead" twitter: "SanneGrinovero" bio: "Dutch, Italian, living in London. Passionate for OSS and the JVM, having fun with Quarkus while leading the Hibernate team at Red Hat." +leydenquarkus: + name: "Maria Arias de Reyna Dominguez, Andrew Dinn, Sanne Grinovero" + job_title: "OpenJDK and Quarkus team working on Project Leyden at Red Hat" + bio: "Maria Arias de Reyna Dominguez is a Java Champion and member of the OpenJDK team at Red Hat; Andrew Dinn is a Distinguished Engineer at Red Hat, and Architect responsible for OpenJDK and GraalVM; Sanne Grinovero is a Java Champion, founding member of the Quarkus project and Architect in the middleware group at Red Hat." dandreadis: name: "Dimitris Andreadis" email: "dimitris@redhat.com" diff --git a/_posts/2024-08-06-quarkus-and-leyden.adoc b/_posts/2024-08-06-quarkus-and-leyden.adoc new file mode 100644 index 00000000000..83924848e2d --- /dev/null +++ b/_posts/2024-08-06-quarkus-and-leyden.adoc @@ -0,0 +1,275 @@ +--- +layout: post +title: 'Project Leyden' +date: 2024-08-06 +tags: leyden +synopsis: 'Project Leyden: exploring its potential for Quarkus users.' +author: leydenquarkus +--- +:imagesdir: /assets/images/posts/leyden + +You might have heard of https://openjdk.org/projects/leyden/[Project Leyden]: an initiative within the OpenJDK project with very ambitious goals. + +When casually namedropping "I'm looking into Project Leyden" with some fellow Java developers these days, I would usually spot some off guard thinking: "Leyden..?", to then offer a hasty save "Ah, right, the project to improve startup times". + +Which is not wrong, as indeed startup times improvements are one of its goals, yet we believe the other stated goals of the project offer even greater potential for our favourite platform and its users. + +== What is Leyden? + +Project Leyden is an initiative from the OpenJDK team. It is an ongoing experiment that is currently being developed by the joint effort of teams from different companies. + +[quote] +____ +The primary goal of this Project is to improve the startup time, time to peak performance, and footprint of Java programs. + +-- Project Leyden, first thing on its project page +____ + +Leyden is a general umbrella project to address slow startup and large footprint. To keep costs down, in all forms such as energy consumption, required hardware resources, and indeed money, it's useful to keep bootstrap times reasonably low, but it's even more effective to reduce the time to peak performance, such as the time it takes for the JVM to "warm up", and reducing the footprint of our applications, such as total memory, has very direct impact. + +Note that the project is evolving rapidly: some of the things explained in this article are evolving while this is written. If you plan on getting involved at a more technical level, follow the development in Jira and the https://mail.openjdk.org/mailman/listinfo/leyden-dev[Leyden mailing list]. + +=== Why it’s interesting to Quarkus + +From a Quarkus perspective, we've done a fair job on all such metrics but we're constantly on the lookout to improve. +That's why Project Leyden got our attention. We're already working with our collagues from the OpenJDK team at Red Hat, who are directly involved in implementing Leyden with the wider OpenJDK group: this blog post today is a collaboration among engineers from different teams. + +Although Quarkus is already doing a lot of work during the Ahead of Time phase to speed up warmup and response time, the enhancements that Leyden is bringing to the table are more related to how the JVM behaves. Complementing both approaches, the advantages we can expect from the combination of Quarkus and Leyden are beyond anything you can find with either of them separated. + +Since the potential for such technological collaboration is strong, the Quarkus and OpenJDK teams are working together on various prototypes and anyone in the Quarkus community would be very welcome to join as well. + +== Refresher on JVM's bootstrap process + +To better understand the scope of the potential improvements, we need to take a step back and discuss how the JVM works today, especially how our application is started and iteratively evolves from interpreting our bytecode to its highest performance mode: running native code which is highly optimized, adapted to the particular hardware, the configuration of the day, and the specific workloads it's been asked to perform. +No other runtime is able to match the JVM on this. + +As we all know, a Java runtime does not directly run Java source code. The content of our JAR file is not executable machine code, but Java bytecode generated from Java source code using the javac compiler. + +A key feature of bytecode is portability, encoding the structure of Java classes and operation of their methods in a machine and operating-system independent format. A Java runtime obeys the type information in the bytecode when laying out Java objects. Execution of a method normally involves interpreting the operations in the method bytecode, although a runtime may also choose to compile method bytecode to equivalent, native machine code and execute the latter directly. + +The unit of delivery for bytecode is a class file, which models a single class. The Java runtime itself provides a host of utility and runtime management classes, as class files embedded in either system jars or jmod files. Applications supplement this with their own class files, usually by appending jars to the classpath or module path. + +Bytecode is delivered class-at-a-time to allow the runtime to load classes _lazily_: i.e. the runtime will only lookup, verify and consume a class file when that class's definition is required to proceed with execution. + +Lazy loading is what allows Java to be a dynamic language -- i.e. one where the code that is included in the program can be decided at runtime. That might include loading classes from jars identified at runtime, possibly loaded via the network. Alternatively, it might include generating class bytecode at runtime, as is done with proxy classes or service provider auxiliary classes. + +=== Just in Time (JIT) and Ahead of Time (AOT) + +Another name to describe Java's lazy loading is 'Just in Time (JIT). JIT is a well known term used to describe the operation of Java's runtime compilers. What is less well known is that it has a much wider use. Many other operations performed by the JVM are done lazily at runtime or 'Just In Time'. + +An alternative to doing things 'Just in Time' (JIT) is to do them 'Ahead Of Time' (AOT). For example, Graal's Native Image runtime loads and analyses the bytecode of every single class needed by an application, including JDK runtime classes, 'Ahead Of Time' i.e. at image build time. It uses the type and method information encoded in that bytecode to 'Ahead Of Time' compile a complete program that includes code for every method that might possibly be executed by the application. + +Graal Native Image lies at one extreme, everything is done AOT, and the traditional Java runtime model lies at the other extreme, as much as possible is done JIT. However, it is actually possible to mix and match AOT and JIT models of execution in one runtime: re-balancing that AOT vs JIT mix is the goal of the first EA release of project Leyden. + +image::AoT_vs_JiT.svg[Compilation work diagram,float="right",align="center"] + +=== Class Data Sharing (CDS) as a step to AOT Caching + +Indeed, this is not a wholly new idea as far as the OpenJDK runtime is concerned. OpenJDK has supported a hybrid AOT/JIT class loading model for years with CDS. The observation that led to https://docs.oracle.com/en/java/javase/21/vm/class-data-sharing.html[Class Data Sharing (CDS)] being proposed was that most applications load the same classes every time they run, both JDK classes during JDK bootstrap and application classes during application startup and warmup. + +Loading requires locating a class bytecode file, possibly calling out to a Java ClassLoader, parsing the bytecode then building a JVM-internal model of the class. This internal model unpacks the information packed into the bytecode into a format that enables fast interpreted or compiled execution. If this loading work could be done once and the resulting class model efficiently reused on subsequent runs, then that would save time during startup and warm up. + +Initially CDS optimized loading for a large set of core JDK classes. It worked by booting the JVM and dumping the class model for all classes loaded during startup into an archive file laid out in memory format. The resulting JDK module, class, field, and method graph can then be quickly remapped into memory next time the JVM runs. Loading a class that is present in the archive involves a simple lookup in the AOT class model. Loading a class not present in the archive requires the normal JIT steps of bytecode lookup, parsing and unpacking i.e. CDS implements a hybrid JIT/AOT execution model. + +A default CDS archive for JDK runtime classes has been shipped with every JVM release since JDK17, halving JDK startup time. Improvements were made to CDS to allow application classes to be include in a CDS archive after executing a short application training run. The resulting mixed AOT/JIT operation can provide significant improvements to application startup and warmup times, depending on how well the training run exercises application code. So, selective JIT/AOT operation is not some new thing. + +The goal of Project Leyden is extending the AOT vs JIT trade-off from class loading (as done by CDS) to other JIT operations in the JVM, the lazy linking that normally happens during interpreted execution and the lazy compilation and recompilation that happens when methods have been executed enough times to justify the cost of compilation. + + +== AOT vs JIT Linkage + +Linking of classes is another operation that the JVM does lazily. When class bytecode is processed the class is directly linked to its owning module and its owned methods and fields. JIT linkage connects elements of each independent, linked class sub-graph into a fully connected graph where elements from different (class or module) files cross-reference each other. + +Clearly, loading and linking proceed recursively. As one example, every class (except Object) needs to be linked to its super class. Super linkage cannot complete without ensuring the super class is loaded. Indeed, if the super's bytecode cannot be found or is not valid (say it identifies an interface not a class) then a linkage error may occur. Likewise, a static field get/put or a new operation the bytecode of some method can only be linked after loading of the target class for the get/put or new. + +Linking is sometimes, but not always, done lazily. Indeed, it is necessary to do some linkage lazily in order to allow loading also to be lazy, otherwise the whole class graph would end up being linked as soon as the main routine was entered. Super linkage is always done eagerly at the point where the subclass has just been loaded. That is because it is not possible to use a subclass to create instances or execute methods without knowing how the superclass is defined. By contrast, field and method linkage is done lazily. In these cases linkage happens as a side-effect of execution. When a method executes a field get/put or method invoke bytecode for the first time the target field or method is looked up via its owner class, loading it if necessary. The field type or method signature is checked for consistency and details of where to find the field or how to call the method are cached, allowing the next execution of the bytecode to bypass the linkage step. + +As with lazy loading, this lazy approach results in almost the exact same linkage being established on every run. The time spent stopping and restarting execution to lazily connect the class graph comprises a noticeable percentage of JDK startup, application startup and application warm up (time to peak running). We could speed up startup and, more crucially, warm up time if we could pre-compute this linkage and avoid the need to establish it at runtime. + +=== Synergy with Quarkus + +Loading and linking of classes is an important step in the warm up of the application because it involves searching through the whole classpath for all classes and objects referenced by the bytecode the JVM is going to run. By default, this is done as a lazy operation because loading and linking all existing classes in the classpath would not only require a bigger memory footprint, but also a bigger warm up time. This is why the JVM only compiles and links the bytecode that is going to be used. + +This is a process that Quarkus already speeds up by, among other strategies, aggressively reducing the set of classes included in the classpath, so the search for matches is faster. The search for classes is also accelerated by indexes which Quarkus can generate when it fully analyzes the application at build time. But it is still a heavy operation that is difficult to execute ahead of time, before we know what is going to be run and how. Quarkus might be able to provide some additional hints to the linker in the future. + +The first improvement Leyden is offering to improve startup time is to upgrade the AOT model originally develped as part of the CDS project to encompass not just pre-loading of classes but also pre-linking, as described in https://openjdk.org/jeps/8315737[JEP Ahead-of-Time Class Linking]. + +An AOT archive can be generated during a training run that bootstraps the JVM and, optionally, executes application-specific code. +As with CDS the AOT archive stores a class graph for all classes loaded during the training run in a format that allows it to be quickly remapped on a subsequent run. The stored graph also includes any linkage information established by code executed during the training run. Pre-cached links avoid the need to stop and start execution to perform linkage on subsequent runs. + +Remember that the training run enables some of the loading and linking to be done AOT but that anything not trained for will still be performed via the regular JIT process: the AOT approach is not required to be applied comprehensively, so that the JVM can fallback to the regular loading system for the use cases which can not benefit from AOT processing. + + +=== JIT vs AOT Compilation + +Another well-known lazy operation the JVM performs is JIT (runtime) compilation. Method bytecode is normally interpreted, but the JVM will lazily translate bytecode to equivalent machine code. +Since generating optimal machine code is an expensive operation, it performs this compilation task selectively, only bothering to compile methods that have been invoked quite a few times. + +JIT compilation is also 'adaptive' i.e. the JVM will also lazily upgrade compiled code after it has been executed very many times, using a different 'tier' or level of compilation: + + . An initial tier 1 compile runs quickly, generating code that is only lightly optimised using profile information gathered during interpretation. + . A tier 2 recompile will instrument the code to track more details about control flow. + . Tier 3 compilation adds further instrumentation that records many more details about what gets executed, including with what type of values. + . Finally a tier 4 compilation uses the gathered profile information to perform a great deal of optimization. + +This final stage of compilation can take a very long time so compilation above tier 1 only happens for a small subset of very frequently executed methods. + +Sometimes, the code is compiled with substantial optimisations based on assumptions extrapolated from the profiling data. +In such cases, the compiler will make an optimistic assumption about a condition to be consistently true in the future yet include an efficient check to verify the assumption during execution so that the semantics of the program are not affected in case this educated guess eventually turns out to be false; when this is detected, the code is de-optimised, returning at a previous tier of compilation and the profiling data is adjusted, so that it will eventually be recompiled with better information. +Essentially, some parts of code might get recompiled multiple times and occasionally revert to a lower tier: it's an highly dynamic process. + +Peak optimization is reached when most of the running code is compiled at the highest tier, and background compilation activities become very rare or, ideally, none at all. + +Compiling code for peak performance also requires quite some resources, so performing this work ahead of time can also save precious CPU cycles during the application bootstrap, and can manifest in substantial memory savings as well. + +But there are some limitations on what we can optimise before runtime just by examining the bytecode. For example, extensive use of reflection prevents the compiler from predicting which symbols will be loaded, linked, and most used at runtime. + +The Leyden project has already sucessfully prototyped shifting the work of method compilation from JIT to AOT. Execution and compilation of methods is tracked during the training run. At the end of the run any associated profiling information and compiled code for the method are saved to the AOT archive, allowing them to be quickly mapped back into memory and reused when the application is next run. + +As with AOT loading and linking, the training run enables some of the work of profiling and compiling to be done AOT but allows anything not trained still to be compiled via the regular JIT compilation process. Note that method code does not need to have been compiled at the highest tier in order to be saved. Also, when code compiled at a lower tier is restored it can still be recompiled at a higher level. + +It can also be deoptimized and re-optimized to adapt to different runtime conditions, just as with code compiled in the current runtime. So, the use of AOT compilation is fully integrated into OpenJDK's adaptive, dynamic compilation and recompilation model: even if some assumptions made during AOT compilation turn out to be suboptimal, the just-in-time compiler can intervene at runtime and improve the code with the new information. + +== Current status of Project Leyden + +There are already experimental https://jdk.java.net/leyden/[early-access builds of Leyden] that can be tested based on https://openjdk.org/jeps/8315737[this draft JEP about Ahead-of-Time Class Linking]. With the https://www.youtube.com/watch?v=NlJK5BKXtHI[Leyden Project], the training run idea has been extended to a wider range of data structures embedded in the new AOT cache. Now the training data contains: + + - Class file events with historical data (Classes loaded and linked, Compilations) + - Resolution of API points and indy (stored in constant pool images in the AOT archive). If you have lambdas in your code, they are captured here. + - Pre-created constant objects in the Java heap (String and Class constants) + - Execution profiles and some compiled native code (all tiers) + +=== Some known limitations + +This is an experimental project being developed by multiple teams having different approaches and focuses. Limitations explained here are being worked on at the time of writing this blog post. + +One of the main issues is that functionality is currently only available for x86_64 and AArch64 architectures at the moment. + +Also, current developments rely on a flat classpath. If the application is using custom classloaders, then it may not benefit as much as it could as it may miss caching many classes. + +The same happens if the application is intensively using reflection. Quarkus avoids reflection whenever possible, preferring to resolve reflective calls at build time as well - so there’s a nice synergy right there. + +However Quarkus in “fast-jar” mode, which is the default packaging mode, will use a custom classloader which currently would get in the way of some Leyden optimisations. One could use a different packaging mode in Quarkus to get more prominent benefits from Leyden, but doing so would disable other Quarkus optimisations, so the comparison wouldn’t be entirely fair today. +We hope to work on improvements in this area to have all possible benefits, combined. + +The focus on these first early releases has been on bootstrap times. There are measurable, significant startup time improvements, due to AoT loading and linking. In some cases, these improvements on startup time have worsened the memory footprint of some applications. That’s an already known issue that is being worked on, and the expected outcome is to improve memory footprint as well, so we would suggest not worrying too much about total memory consumption at this stage. + +Since the AOT archives include machine specific optimisations such as the native code generated by the C2 compiler, the training run and the production run must be done on the same type of hardware and JDK versions; it also requires using the same JAR-based classpaths and the same command line options. + +Although you can use a different Main class for running the application, for example a test class that simulates usage. + +=== What is on the roadmap for Leyden? + +There’s still work to be done regarding classes that can’t be loaded and linked in AoT with the current implementation. For example, classes loaded using a user-defined class loader. There’s also room to improve the way the training runs are made, maybe allowing the user to tweak the results to influence decisions. + +Currently, the https://bugs.openjdk.org/browse/JDK-8326035[Z Garbage Collector] does not support AOT object archiving. There is an active effort to make sure all Garbage Collectors are compatible with these enhancements. + +There are also other things planned in the roadmap for Leyden, like adding condensers. https://openjdk.org/projects/leyden/notes/03-toward-condensers[Condensers] will be composable transformers of the source code in AoT that modify the source code optimising it. Each developer will be able to define a pipeline of condensers that improves their source code before compiling it into bytecode; this is very interesting to the Quarkus team but condensers aren’t available yet. + +The OpenJDK team is working on adding more complete code save and restore to the AOT cache to avoid that first compilation for trained data, by just loading the compiled code directly from the cache; our colleagues from Red Hat’s OpenJDK team are directly involved in implementing this. This could include, among others, auxiliary code used to interface compiled code to runtime, interpreter or other compiled runtimes. + +== How to play with it + +The first step would be to install one of the early Leden builds that you can find in https://jdk.java.net/leyden/ + +Make sure that you have installed it correctly by running the following command: + +[source, console] +---- +$ java --version +openjdk 24-leydenpremain 2025-03-18 +OpenJDK Runtime Environment (build 24-leydenpremain+2-8) +OpenJDK 64-Bit Server VM (build 24-leydenpremain+2-8, mixed mode, sharing) +---- + +Go to the application you want to test Leyden with and start a first training run: + +[source, console] +---- +$ java -XX:CacheDataStore=quarkusapp.cds -jar $YOUR_JAR_FILE +---- + +This will generate the archive files with all the profiling information needed to speed up the production run. + +Now that we have them, we can run our application using the Leyden enhancements: + +[source, console] +---- +$ java -XX:CacheDataStore=quarkusapp.cds -XX:+AOTClassLinking -jar $YOUR_JAR_FILE +---- + +== Potentially needed workarounds + +Since it’s early days for the Leyden project, there are some known limitations. The following instructions shouldn’t be necessary for the final versions but you might need them today. + +=== Force the use of G1GC + +To benefit from the natively compiled code in AOT archives, the garbage collector used at runtime needs to match the same garbage collector used when you recorded the AOT archives. + +Remember that the JVM’s default choice of garbage collector is based on ergonomics; normally this is nice but it can cause some confusion in this case; for example if you build on a large server it will pick G1GC by default, but then when you run the application on a server with constrained memory it would, by default, pick SerialGC. + +To avoid this mismatch it’s best to pick a garbage collector explicitly; and since several AOT related optimisations today only apply to G1, let’s enforce the use of G1GC. + +Force using G1GC: + +[source, console] +---- +-XX:+UseG1GC +---- + +N.B. you need to use this consistently on both the process generating the AOT archives and the runtime. + +=== Force the G1 Region sizes + +As identified and reported by the Quarkus team to our colleagues working on Project Leyden, beyond enforcing a specific garbage collector one should also ensure that the code stored in AOT archives is being generated with the same G1 region sizes as what’s going to be used at runtime, or one risks segmentation faults caused by it wrongly identifying regions. +See https://bugs.openjdk.org/browse/JDK-8335440 for details, or simply set: + +Configure G1HeapRegionSize explicitly: + +[source, console] +---- +-XX:G1HeapRegionSize=1048576 +---- + +N.B. you need to use this consistently on both the process generating the AOT archives and the runtime. + +=== Failure to terminate in containers + +This issue has already been resolved, but in case you’re using an older version of project Leyden and it fails to exit on regular container termination, you might be affected by https://bugs.openjdk.org/browse/JDK-8333794[JDK-8333794]. + +Workaround for JDK-8333794: + +[source, console] +---- +-Djdk.console=java.basebroken +---- + +== Will Leyden replace GraalVM's native-image capabilities? + +The short answer is no. + +If you want the absolute smallest footprint and ensure that absolutely no "dynamic" adaptations happen at runtime, GraalVM native images are the way to go. Just think about it: to support the dynamic aspects that the JVM normally provides, +even in very minimal form, you would need some code which is able to perform this work, and some memory and some computational resources to run such code and adapt your runtime safely; this is a complex feature and will never be completely free, even in the case Leyden evolved significantly beyond the current plans. + +The architecture of Quarkus enables developers to define an application in strict "closed world" style, and this approach works extremely well in combination with GraalVM native images, but the Quarkus design works indeed very well on the bigger, dynamic JVMs as well. + +The ability that Quarkus offers to created a closed world application doesn't imply that you should necessarily be doing so; in fact there are many applications which could benefit from a bit more dynamism, some more runtime configurability or auto-adaptability, and Quarkus also allows to create such applications while still benefitting from very substantial efficiency improvements over competing architectures, and even over competing runtimes and languages. + +We're very excited by Project Leyden as it allows to substantially improve bootstrap times, warmup times, and overall costs even for the "regular" JVM, so retaining all the benefits of a dynamic runtime and an adaptative JIT compiler, and this will be a fantastic option for all those applications for which a fully AOT native image might not be suitable: you'll get some of the benefits from native-image (not all of them) but essentially for free, at no drawbacks. + +We also hope it will bring better defined semantics in regards to running certain phases “ahead of time” (or later); there is a very interesting read on this topic by Mark Reinhold: “Selectively Shifting and Constraining Computation” ; from a perspective of Quarkus developers, we can confirm that improvements in the language specification in this area would be very welcome, and also improve the quality and maintainability of applications compiled with GraalVM native-image(s). + +For these reasons, Quarkus will definitely not deprecate support for native images; it's more plausible that, eventually, the "full JVM" will always be benefitting from Leyden powered improvements, and as usual we'll work to make these benefits work in synergy with our architecture, and at minimal effort for you all. + +Essentially both the JVM and the native-image options are bound to benefit from this initiative. It's a great time to be a Java developer! + + +== How can I make sure this will work for me? + +The best way to make sure your application benefits from Leyden is to start experimenting early, be involved in the development. It would be great to add real-world feedback from a perspective of Quarkus users. + +If you spend some time testing your application with the https://jdk.java.net/leyden/[early-access builds of Leyden], and https://bugs.openjdk.org/browse/JDK-8332177?jql=issuetype%20%3D%20Bug%20AND%20status%20%3D%20Open%20AND%20labels%20%3D%20leyden[reporting any bugs] or weird behaviour the developers will take your specificities into account. + +The OpenJDK issue tracker isn’t open to everyone, but you’re also very welcome to provide feedback on our https://quarkus.io/discussion/[Quarkus channels]; we can then relay any suggestions to our colleagues who are directly working on project Leyden. +You can also use the https://mail.openjdk.org/mailman/listinfo/leyden-dev[Leyden mailing list]. + diff --git a/assets/images/posts/leyden/AoT_vs_JiT.svg b/assets/images/posts/leyden/AoT_vs_JiT.svg new file mode 100644 index 00000000000..5588330117b --- /dev/null +++ b/assets/images/posts/leyden/AoT_vs_JiT.svg @@ -0,0 +1,102 @@ + + + + + + + + + + + + + + + + + Just In Time (JiT) + + Ahead of Time (AoT) + + Developer's Machine + + + + + + + Java Virtual Machine (JVM) + + + + + + + + + + Compile + (static code) + + + + + + + + + + Pre-main + (Leyden) + + + + + + + ByteCode + (JAR) + + + + + + + + + + Compile, Link, + and Optimize + on the fly + + + + + + + + + + Run + + + + + + + Training Run Data + (CDS) + + + + + + + + + + + + + +