Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hibernate-search] Introduce Hibernate Search framework and implement indexing page #6218

Open
wants to merge 30 commits into
base: hibernate-search
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
f94697a
Add HibernateSearch dependencies to DataManagement module
solth Aug 6, 2021
43dbc44
Declare Hibernate Search version in root POM, and bump to 6.2.4.Final
matthias-ronge Sep 3, 2024
10ebf2f
Add HibernateSearch annotations to base indexed classes
solth Nov 1, 2021
adcd4a1
Add index names, add 'Indexed' annotation
solth Nov 2, 2021
d1a5c97
Add annotations for complex fields
solth Nov 3, 2021
e8c841a
Add annotations for Docket, Filter, Ruleset and Workflow
solth Nov 4, 2021
6c28a24
Show the indexing page if the search server is available
matthias-ronge Sep 3, 2024
40e5249
Remove Create Mapping and Delete Index buttons (henceforth implied)
matthias-ronge Sep 4, 2024
e97eba0
Remove button to index remaining - not supported by Hibernate Search
matthias-ronge Sep 4, 2024
26280cf
Index all objects of given 'objectType' with massIndexer
IkramMaalej Nov 8, 2021
faeed43
Re-implement indexing page
matthias-ronge Sep 4, 2024
40ef313
Fix checkstyle
matthias-ronge Sep 4, 2024
11bcf85
Improve wording, add Javadoc
matthias-ronge Sep 4, 2024
caf4bae
Don't show a total of 0 objects when starting indexing
matthias-ronge Sep 5, 2024
5b2e0ef
Returns result processing to the calling class
matthias-ronge Sep 5, 2024
d6ae013
Remove test for mapping - created transparently
matthias-ronge Sep 5, 2024
24a2005
Set number of database objects
matthias-ronge Sep 6, 2024
5fa7f85
Add template count
matthias-ronge Sep 6, 2024
732997c
Bring OpenSearch background instance for tests
matthias-ronge Sep 6, 2024
ea5f0e3
Add MockDatabase index to Kitodo - DataManagement
matthias-ronge Sep 6, 2024
6c0eecc
Fix test
matthias-ronge Sep 9, 2024
5fa2d88
Increase timeout (slow laptop)
matthias-ronge Sep 9, 2024
60bcacf
Fix problems
matthias-ronge Sep 9, 2024
d97ac6a
Fix search for ID
matthias-ronge Sep 9, 2024
9c20c89
Log all queries
matthias-ronge Sep 9, 2024
0c8374c
Add missing file for tests
matthias-ronge Sep 9, 2024
afd9ae8
Fix tests
matthias-ronge Sep 10, 2024
3369fa3
Add Hibernate Search config file to Selenium resources
matthias-ronge Sep 10, 2024
963e1c7
Remove unused imports
matthias-ronge Sep 11, 2024
c1bbea7
Add tasks to processes to enable sorting by sortHelperStatus
matthias-ronge Sep 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions Kitodo-API/src/main/java/org/kitodo/config/KitodoConfig.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@
package org.kitodo.config;

import java.io.File;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URL;
import java.nio.file.Paths;
import java.util.NoSuchElementException;
import java.util.Optional;
Expand Down Expand Up @@ -237,6 +239,27 @@ public static Optional<String> getOptionalString(ParameterInterface key) {
}
}

/**
* Returns the URL of the search server.
*
* @return the URL
* @throws MalformedURLException
* if an unknown protocol, or the port is a negative number
* other than -1
*/
public static URL getSearchServerUrl() throws MalformedURLException {
String host = getParameter("elasticsearch.host", "localhost");
int port = getIntParameter(new ParameterInterface() {
@Override
public String getName() {
return "elasticsearch.port";
}
}, 9200);
String protocol = getParameter("elasticsearch.protocol", "http");
String path = getParameter("elasticsearch.path", "/");
return new URL(protocol, host, port, path);
}

/**
* Returns the selected URI from the configuration file. Throws a
* {@code NoSuchElementException} if no such parameter exists.
Expand Down
3 changes: 3 additions & 0 deletions Kitodo-DataManagement/hibernate.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
hibernate.search.enabled=true
hibernate.search.backend.hosts=localhost:9200
hibernate.search.backend.protocol=http
Comment on lines +1 to +3
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question must this content not be added to the existing hibernate.cfg.xml file or did we need two configuration files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, this is the configuration file for the Hibernate Search framework. I would be surprised if we could mix the two configurations.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean, that everyone must run ElasticSearch / Opensearch on localhost and port 9200? If so I'm unable to do this in my development system nor on a productive environment.

21 changes: 21 additions & 0 deletions Kitodo-DataManagement/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,16 @@
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jcache</artifactId>
</dependency>
<dependency>
<groupId>org.opensearch</groupId>
<artifactId>opensearch</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.opensearch.plugin</groupId>
<artifactId>transport-netty4-client</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-annotations</artifactId>
Expand Down Expand Up @@ -121,6 +131,17 @@
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</dependency>
<!-- Hibernate Search dependencies -->
<dependency>
<groupId>org.hibernate.search</groupId>
<artifactId>hibernate-search-mapper-orm</artifactId>
<version>${hibernate-search.version}</version>
</dependency>
<dependency>
<groupId>org.hibernate.search</groupId>
<artifactId>hibernate-search-backend-elasticsearch</artifactId>
<version>${hibernate-search.version}</version>
</dependency>
</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import javax.persistence.MappedSuperclass;

import org.hibernate.Hibernate;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.kitodo.data.database.persistence.BaseDAO;

/**
Expand All @@ -34,6 +35,7 @@ public abstract class BaseBean implements Serializable {

@Id
@Column(name = "id")
@GenericField
@GeneratedValue(strategy = GenerationType.IDENTITY)
protected Integer id;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,22 @@
import javax.persistence.Column;
import javax.persistence.MappedSuperclass;

import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;

/**
* This bean contains properties common for Template and Process.
*/
@MappedSuperclass
public abstract class BaseTemplateBean extends BaseBean {

@GenericField
@Column(name = "title")
protected String title;

@Column(name = "creationDate")
protected Date creationDate;

@GenericField
@Column(name = "sortHelperStatus")
private String sortHelperStatus;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
import javax.persistence.ManyToMany;
import javax.persistence.Table;

import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.Indexed;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.IndexedEmbedded;
import org.kitodo.data.database.enums.BatchType;
import org.kitodo.data.database.persistence.BatchDAO;

Expand All @@ -38,13 +41,15 @@
* multi-journal binding unit.
*/
@Entity
@Indexed(index = "kitodo-batch")
@Table(name = "batch")
public class Batch extends BaseBean {

/**
* The batch title. Using titles for batches is optional, the field may be
* {@code null}. If so, the ID will be shown to the user instead.
*/
@GenericField
@Column(name = "title")
private String title;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,14 @@
import javax.persistence.OneToMany;
import javax.persistence.Table;

import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.kitodo.data.database.persistence.ClientDAO;

@Entity
@Table(name = "client")
public class Client extends BaseBean {

@GenericField
@Column(name = "name")
private String name;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import javax.persistence.ManyToOne;
import javax.persistence.Table;

import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.kitodo.data.database.enums.CommentType;

@Entity
Expand All @@ -31,13 +32,15 @@ public class Comment extends BaseBean {
* The field message holds the comment message.
*/
@Column(name = "message", columnDefinition = "longtext")
@GenericField
private String message;

/**
* The field type holds the comment type.
*/
@Column(name = "type")
@Enumerated(EnumType.STRING)
@GenericField
private CommentType type;

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,32 @@
import javax.persistence.ManyToOne;
import javax.persistence.Table;

import org.hibernate.search.mapper.pojo.automaticindexing.ReindexOnUpdate;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.Indexed;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.IndexedEmbedded;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.IndexingDependency;

@Entity
@Indexed(index = "kitodo-docket")
@Table(name = "docket")
public class Docket extends BaseBean {

@GenericField
@Column(name = "title")
private String title;

@GenericField
@Column(name = "file")
private String file;

@GenericField
@Column(name = "active")
private Boolean active = true;

@ManyToOne
@IndexedEmbedded(includePaths = {"id", "name"})
@IndexingDependency(reindexOnUpdate = ReindexOnUpdate.SHALLOW)
@JoinColumn(name = "client_id", foreignKey = @ForeignKey(name = "FK_docket_client_id"))
private Client client;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,28 @@
import javax.persistence.ManyToOne;
import javax.persistence.Table;

import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.Indexed;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.IndexedEmbedded;

/**
* Filter bean.
*/
@Entity
@Indexed(index = "kitodo-filter")
@Table(name = "filter")
public class Filter extends BaseBean {

@GenericField
@Column(name = "value", columnDefinition = "longtext")
private String value;

@GenericField
@Column(name = "creationDate")
private Date creationDate;

@ManyToOne
@IndexedEmbedded(includePaths = {"id"})
@JoinColumn(name = "user_id", foreignKey = @ForeignKey(name = "FK_filter_user_id"))
private User user;

Expand Down
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is an @Indexed(index = "kitodo-folder") annotation not missing like in the other bean files?

Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import javax.persistence.Table;
import javax.persistence.Transient;

import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.kitodo.api.imagemanagement.ImageManagementInterface;
import org.kitodo.config.ConfigMain;
import org.kitodo.data.database.enums.LinkingMode;
Expand Down Expand Up @@ -104,6 +105,7 @@ public class Folder extends BaseBean {
* contents of this folder will be linked.
*/
@Column(name = "fileGroup")
@GenericField
private String fileGroup;

/**
Expand All @@ -130,12 +132,14 @@ public class Folder extends BaseBean {
* @see org.kitodo.config.xml.fileformats.FileFormatsConfig
*/
@Column(name = "mimeType")
@GenericField
private String mimeType = "image/jpeg";

/**
* The path to the folder in the process directory of each processes.
*/
@Column(name = "path")
@GenericField
private String path = "";

/**
Expand All @@ -151,6 +155,7 @@ public class Folder extends BaseBean {
* replaced before concatenation.
*/
@Column(name = "urlStructure")
@GenericField
private String urlStructure;

/**
Expand Down
Loading