EPrints4 Core Roadmap - Revision history

Kgoetz: Change category Developer to Contribute

2018-09-11T23:53:55Z

Change category Developer to Contribute

← Older revision		Revision as of 23:53, 11 September 2018
Line 1:		Line 1:
	[[Category:Eprints4]]		[[Category:Eprints4]]
	[[Category:~~Developer~~]]		[[Category:Contribute]]

	=EPrints4 Core Development=		=EPrints4 Core Development=

Kgoetz: adding categories

2018-08-30T06:36:44Z

adding categories

← Older revision		Revision as of 06:36, 30 August 2018
Line 1:		Line 1:
			[[Category:Eprints4]]
			[[Category:Developer]]

	=EPrints4 Core Development=		=EPrints4 Core Development=

Sf03r@ecs.soton.ac.uk: /* Data retrieval and searching */

2014-02-07T11:12:42Z

Data retrieval and searching

← Older revision		Revision as of 11:12, 7 February 2014
Line 89:		Line 89:

	* an internal mechanism for retrieving data straight from the database, for instance:		* an internal mechanism for retrieving data straight from the database, for instance:

	** all data objects belonging to a dataset		** all data objects belonging to a dataset
	** all data objects owned by a user		** all data objects owned by a user
	** generally the retrieval of data objects based on an EXACT constraint or set of constraints.		** generally the retrieval of data objects based on an EXACT constraint or set of constraints.

	** possibility to order the data objects by a given meta field (i.e. a DB column).		** possibility to order the data objects by a given meta field (i.e. a DB column).

	* a user search optimised for helping users to find content they're interested to. This will likely be done by Xapian. Because this concept can become difficult to implement, it is better to use a specialist tool such as Xapian to do it. Includes the following scenarios:		* a user search optimised for helping users to find content they're interested to. This will likely be done by Xapian. Because this concept can become difficult to implement, it is better to use a specialist tool such as Xapian to do it. Includes the following scenarios:

	** retrieval of all data objects belonging to a dataset (via the Xapian indexes which may be out-of-date or incomplete since objects are indexed by the indexer in a asynchronous manner)		** retrieval of all data objects belonging to a dataset (via the Xapian indexes which may be out-of-date or incomplete since objects are indexed by the indexer in a asynchronous manner)
	** parsing complex user queries using e.g. boolean operators		** parsing complex user queries using e.g. boolean operators

Sf03r@ecs.soton.ac.uk: /* Data set / Data objects */

2014-02-07T11:12:28Z

Data set / Data objects

← Older revision		Revision as of 11:12, 7 February 2014
Line 71:		Line 71:

	* DataSet/Objects are collections of metadata fields and offer an API for easily manipulating objects/data.		* DataSet/Objects are collections of metadata fields and offer an API for easily manipulating objects/data.

	* Dataset offers a number of properties to handle common fields/concepts in a consistent manner across different datasets. For instance the "lastmod" property which is an internal field always set to when the object was last modified or "history" which keeps track of the modifications of the object. Considered properties:		* Dataset offers a number of properties to handle common fields/concepts in a consistent manner across different datasets. For instance the "lastmod" property which is an internal field always set to when the object was last modified or "history" which keeps track of the modifications of the object. Considered properties:

	** "states": the graph of states for that object. Similar to "eprint_status" for "eprint" objects in EPrints2/3. Automates and generalises state transfer for objects.		** "states": the graph of states for that object. Similar to "eprint_status" for "eprint" objects in EPrints2/3. Automates and generalises state transfer for objects.

	** "revision": keep the revision number which is incremented each time that object is modified (default: ON)		** "revision": keep the revision number which is incremented each time that object is modified (default: ON)

	** "acl": enables the ACL related fields and permission checking (TODO)		** "acl": enables the ACL related fields and permission checking (TODO)

	** "cache": enables the use of memcached (default: ON if globally enabled) - if enabled, keep a copy of data objects in memory rather than reading objects from DB.		** "cache": enables the use of memcached (default: ON if globally enabled) - if enabled, keep a copy of data objects in memory rather than reading objects from DB.

	** "read-only": makes that data-set read only and forbids any modification/deletion (default: OFF)		** "read-only": makes that data-set read only and forbids any modification/deletion (default: OFF)

	** "history": keeps an XML diff of the modifications made to that object. (TODO)		** "history": keeps an XML diff of the modifications made to that object. (TODO)

	** "lastmod": keeps a last modified field		** "lastmod": keeps a last modified field

	** "datestamp": the date at which the object was created		** "datestamp": the date at which the object was created

	* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)		* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)

Sf03r@ecs.soton.ac.uk: /* Data Representation */

2014-02-06T16:10:45Z

Data Representation

← Older revision		Revision as of 16:10, 6 February 2014
Line 91:		Line 91:

	* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)		* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)

			Question: EPrints2/3 does not offer a generalised manner to reference a sub-dataset: "archive" is used as a short-cut for "eprint:archive" but because the dataset naming is global you can't re-use "archive" for other datasets. So we need a common way to reference a dataset. eprint:inbox? eprint/inbox? eprint_inbox?

	==Data retrieval and searching==		==Data retrieval and searching==

Sf03r@ecs.soton.ac.uk: /* CLI */

2014-02-06T16:00:13Z

CLI

← Older revision		Revision as of 16:00, 6 February 2014
Line 122:		Line 122:

	==CLI==		==CLI==

			* Use of a SYSTEM user for CLI operation/scripts? If not, ACL type of checks must online be done/enabled for Web requests

	==Misc==		==Misc==

Sf03r@ecs.soton.ac.uk at 15:59, 6 February 2014

2014-02-06T15:59:20Z

← Older revision		Revision as of 15:59, 6 February 2014
Line 120:		Line 120:

	* Requires the creation and management of indexes (because it is the indexes which are searched)		* Requires the creation and management of indexes (because it is the indexes which are searched)

			==CLI==

			==Misc==

Sf03r@ecs.soton.ac.uk: /* Data retrieval */

2014-02-06T15:58:48Z

Data retrieval

← Older revision		Revision as of 15:58, 6 February 2014
Line 92:		Line 92:
	* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)		* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)

	==Data retrieval==		==Data retrieval and searching==

			I think there are two slightly different aspects of data retrieval:

			* an internal mechanism for retrieving data straight from the database, for instance:

			** all data objects belonging to a dataset
			** all data objects owned by a user
			** generally the retrieval of data objects based on an EXACT constraint or set of constraints.

			** possibility to order the data objects by a given meta field (i.e. a DB column).

			* a user search optimised for helping users to find content they're interested to. This will likely be done by Xapian. Because this concept can become difficult to implement, it is better to use a specialist tool such as Xapian to do it. Includes the following scenarios:

			** retrieval of all data objects belonging to a dataset (via the Xapian indexes which may be out-of-date or incomplete since objects are indexed by the indexer in a asynchronous manner)
			** parsing complex user queries using e.g. boolean operators

			One question is how to label these two slightly different searching mechanism?

			===Internal Data retrieval===

			* Only supports EXACT matching or other SQL-derived tests and operators e.g. NOT NULL (aka is_set)
			* Tightly linked to the concept of EPrints::List
			* May be used on the UI to display lists of items owned by a user ETC but shouldn't be used in real user searches

			===User Searches===

			* Requires the creation and management of indexes (because it is the indexes which are searched)

Sf03r@ecs.soton.ac.uk at 15:47, 6 February 2014

2014-02-06T15:47:22Z

← Older revision		Revision as of 15:47, 6 February 2014
Line 91:		Line 91:

	* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)		* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)

			==Data retrieval==

Sf03r@ecs.soton.ac.uk at 15:42, 6 February 2014

2014-02-06T15:42:48Z

← Older revision		Revision as of 15:42, 6 February 2014
Line 34:		Line 34:

	Filters must be called using Trigger(s) or similar mechanism to avoid having hard-coded modules in the Request handler. It is then easy to select, add, remove filters at run-time.		Filters must be called using Trigger(s) or similar mechanism to avoid having hard-coded modules in the Request handler. It is then easy to select, add, remove filters at run-time.

			==Database==

			* Must contain any SQL handling. Custom SQL in scripts is bad and shows lacks in the DB API. This must handle the usual operations required by the system such as data retrieval, modification etc and any DB optimisation (commonly known as hacks).

			* The default DB engine is likely to be MySQL/InnoDB with support for transactions.

			* The DB layer is "very" low-level and doesn't check security concerns (e.g. can the user modifies that object?). This must be handled by the above layer, usually DataObj and MetaField.

			TODO: an audit of the current DB code to remove un-needed/deprecated methods, to re-organise the required existing methods.


			==Data Representation==

			This defines properties for data objects (for example the "name" of a user) and offers a layer between the raw data (handled by Database) and the application/views using the data (e.g. CRUD, "the" UI, Export/Import plug-ins etc).

			Important: data objects and metadata fields do not know how to render themselves on a UI. This is one of the main design aspect of EPrints4 as to remove dependencies of the UI on the data model.

			===Metadata fields===

			* make sure the data is valid, according to the type's constraints. For instance, you cannot store a string into an integer

			* maps any higher level type (e.g. URL) to a low-level DB type (e.g. varchar)

			* must provide a clean and clear API to easily modify data (set_value, add_value etc)

			* the type "file" whilst not being a piece of metadata directly is nonetheless a valid field - it is the only type that links data to a stored file (rather than being purely DB data).

			* XML/SAX handlers are likely to be kept at the Meta-Field level (because this makes sense)

			* aspects such as indexing, generating order values are more likely to be included in the Search components (so via Xapian for instance)

			* review of the existing types is needed (40ish types at the moment) and non-core types must be removed (recaptcha, compound...)

			===Data set / Data objects===

			* DataSet/Objects are collections of metadata fields and offer an API for easily manipulating objects/data.

			* Dataset offers a number of properties to handle common fields/concepts in a consistent manner across different datasets. For instance the "lastmod" property which is an internal field always set to when the object was last modified or "history" which keeps track of the modifications of the object. Considered properties:

			** "states": the graph of states for that object. Similar to "eprint_status" for "eprint" objects in EPrints2/3. Automates and generalises state transfer for objects.

			** "revision": keep the revision number which is incremented each time that object is modified (default: ON)

			** "acl": enables the ACL related fields and permission checking (TODO)

			** "cache": enables the use of memcached (default: ON if globally enabled) - if enabled, keep a copy of data objects in memory rather than reading objects from DB.

			** "read-only": makes that data-set read only and forbids any modification/deletion (default: OFF)

			** "history": keeps an XML diff of the modifications made to that object. (TODO)

			** "lastmod": keeps a last modified field

			** "datestamp": the date at which the object was created

			* EPrints::DataObj now contains all required information so that custom datasets do not need to have their explicit implementation (EPrints::DataObj::Foo package)