Download Metrics - Revision history

Pm705 at 13:55, 8 February 2010

2010-02-08T13:55:18Z

← Older revision		Revision as of 13:55, 8 February 2010
Line 1:		Line 1:
			[[Category:Management]]
	=Download Metrics in EPrints=		=Download Metrics in EPrints=

Timbrody: /* Exposing Usage */

2007-05-25T13:58:47Z

Exposing Usage

← Older revision		Revision as of 13:58, 25 May 2007
Line 70:		Line 70:
	Eprints 3 includes a script that can expose access logs through an [http://www.openarchives.org/ OAI-PMH] interface. This interface is located at <tt>/cgi/oai_accesslogs</tt>. The interface contains an <code>EPrints::abort</code> at the top that prevents all access - this is because usage logs are potentially sensitive data: a repository will want to implement some kind of access restriction before providing access to their logs.		Eprints 3 includes a script that can expose access logs through an [http://www.openarchives.org/ OAI-PMH] interface. This interface is located at <tt>/cgi/oai_accesslogs</tt>. The interface contains an <code>EPrints::abort</code> at the top that prevents all access - this is because usage logs are potentially sensitive data: a repository will want to implement some kind of access restriction before providing access to their logs.

	The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) to XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).		The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) into XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).

	{\| border=1 class="wikitable"		{\| border=1 class="wikitable"
	\|+~~accesslog~~/~~eprint~~ OAI comparison		\|+eprint/access OAI comparison
	!		!
	!eprint		!eprint
	!~~accesslog~~		!access
	\|-		\|-
	\|identifier		\|identifier
Line 95:		Line 95:
	\|}		\|}

	An OAI ListRecords request may contain date and set filters, although sets aren't yet supported for access logs. Using the Eprints search API a result set of records is generated that matches the OAI arguments e.g. all access log records made after a certain time. Each record is converted to XML~~, depending on~~ the export plugin requested (via a mapping of the metadataPrefix OAI argument to plugin identifier), and output as an ~~XML document~~.		An OAI ListRecords request may contain date and set filters, although sets aren't yet supported for access logs. OAI date filters consist of optional ''from'' and ''until'' arguments that restrict the resulting data to only those accesses that occurred between the two dates.

			Using the Eprints search API a result set of records is generated that matches the OAI arguments e.g. all access log records made after a certain time. Each record is converted into XML using the export plugin requested (via a mapping of the metadataPrefix OAI argument to plugin identifier) and the result output as an OAI response.

	Currently Eprints supports exporting access log records in OpenURL ContextObject format. To be OAI-compliant an interface should also support Dublin Core (but access logs don't make a lot of sense in DC). The following is our example request, exposed as an XML ContextObject in an OAI response:		Currently Eprints supports exporting access log records in OpenURL ContextObject format. To be OAI-compliant an interface should also support Dublin Core (but access logs don't make a lot of sense in DC). The following is our example request, exposed as an XML ContextObject in an OAI response:

Timbrody: /* Exposing Usage */

2007-05-25T13:54:25Z

Exposing Usage

← Older revision		Revision as of 13:54, 25 May 2007
Line 72:		Line 72:
	The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) to XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).		The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) to XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).

	{\| class="wikitable"		{\| border=1 class="wikitable"
	\|+accesslog/eprint OAI comparison		\|+accesslog/eprint OAI comparison
	!		!

Timbrody: /* Capturing Usage */

2007-05-25T13:54:14Z

Capturing Usage

← Older revision		Revision as of 13:54, 25 May 2007
Line 17:		Line 17:
	The pertinent parts of this entry are:		The pertinent parts of this entry are:

	{\| class="wikitable"		{\| border=1 class="wikitable"
	\|+Fields in a HTTP request.		\|+Fields in a HTTP request.
	\|-		\|-
Line 44:		Line 44:
	In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. A simple heuristic is applied to separate out the requests into three potential types: non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'' field:		In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. A simple heuristic is applied to separate out the requests into three potential types: non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'' field:

	{\| class="wikitable"		{\| border=1 class="wikitable"
	\|+Parts of a page request URL.		\|+Parts of a page request URL.
	\|-		\|-

Timbrody: /* Capturing Usage */

2007-05-25T13:53:19Z

Capturing Usage

← Older revision		Revision as of 13:53, 25 May 2007
Line 47:		Line 47:
	\|+Parts of a page request URL.		\|+Parts of a page request URL.
	\|-		\|-
	!~~/12614/~~		!align="center" \|eprint identifier
	!~~/01/~~		!align="center" \|document identifier
	!~~Semantic_Web_Revisted.pdf~~		!align="center" \|file requested
	\|-		\|-
	\|~~align="center" \|eprint identifier~~		\|/12614/
	\|~~align="center" \|document identifier~~		\|/01/
	\|~~align="center" \|file requested~~		\|Semantic_Web_Revisted.pdf
	\|}		\|}

Timbrody: /* Capturing Usage */

2007-05-25T13:52:16Z

Capturing Usage

← Older revision		Revision as of 13:52, 25 May 2007
Line 42:		Line 42:
	\|}		\|}

	In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. ~~Requests are then split up~~ into three ~~sets,~~ non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'' field.		In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. A simple heuristic is applied to separate out the requests into three potential types: non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'' field:

	{\| class="wikitable"		{\| class="wikitable"
Line 56:		Line 56:
	\|}		\|}

	~~Request~~ without an eprint identifier part are ignored. Requests with no document identifier part are defined as abstract requests. Requests with both eprint and document identifier parts are defined as full-text requests.		Requests without an eprint identifier part are ignored. Requests with no document identifier part are defined as abstract requests. Requests with both eprint and document identifier parts are defined as full-text requests.

	A simple filter is implemented in the capture code to handle requests to multi-file documents (e.g. HTML pages with inlined images). Any full-text requests where the referring web page (see above) is also a request to the same full-text are ignored. Assuming the referring web page is correctly supplied by the web client this will count requests to any part of a document, but ignore multiple requests resulting from complex media.		A simple filter is implemented in the capture code to handle requests to multi-file documents (e.g. HTML pages with inlined images). Any full-text requests where the referring web page (see above) is also a request to the same full-text are ignored. Assuming the referring web page is correctly supplied by the web client this will count requests to any part of a document, but ignore multiple requests resulting from complex media.

Timbrody: /* Storing Usage */

2007-05-25T13:19:18Z

Storing Usage

← Older revision		Revision as of 13:19, 25 May 2007
Line 60:		Line 60:
	A simple filter is implemented in the capture code to handle requests to multi-file documents (e.g. HTML pages with inlined images). Any full-text requests where the referring web page (see above) is also a request to the same full-text are ignored. Assuming the referring web page is correctly supplied by the web client this will count requests to any part of a document, but ignore multiple requests resulting from complex media.		A simple filter is implemented in the capture code to handle requests to multi-file documents (e.g. HTML pages with inlined images). Any full-text requests where the referring web page (see above) is also a request to the same full-text are ignored. Assuming the referring web page is correctly supplied by the web client this will count requests to any part of a document, but ignore multiple requests resulting from complex media.

	==Storing Usage==		==Storing Usage Data==

	Requests are stored in the Eprints ~~database in the~~ ''~~access_log~~'' ~~table~~, see [~~Access_Log~~] for a description of the fields. This is ~~almost a~~ mirror of the normal Apache log file, but by storing the data in a table enables much simpler exposure and re-use of the ~~datab~~.		Requests are stored in the Eprints ''access'' dataset, see [[Access_Object]] for a description of the dataset's fields. This is partially-processed mirror of the normal Apache log file, but storing the data in a table enables much simpler exposure and re-use of the data.

	~~Log data can potentially get very large~~, which ~~may require~~ a ~~different technical approach in future (~~or ~~periodic triaging of~~ the ~~access log data)~~.		The Eprints dataset API provides a unique identifier for every request, which is an incremented integer starting at 1. As a dataset requests can be searched or iterated over using the Eprints API.

	==Exposing Usage==		==Exposing Usage==

Timbrody at 13:13, 25 May 2007

2007-05-25T13:13:10Z

← Older revision		Revision as of 13:13, 25 May 2007
Line 1:		Line 1:
	~~=Download Metrics=~~

	=Download Metrics in EPrints=		=Download Metrics in EPrints=

Line 7:		Line 5:
	Eprints uses the Apache web server and mod_perl (an accelerator for Perl CGI scripts). mod_perl provides hooks to the logging and control features of the Apache server. One of these hooks is to the access logging function. A ''log handler'' is registered with Apache by using the ''[http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler PerlLogHandler]'' pragma e.g.		Eprints uses the Apache web server and mod_perl (an accelerator for Perl CGI scripts). mod_perl provides hooks to the logging and control features of the Apache server. One of these hooks is to the access logging function. A ''log handler'' is registered with Apache by using the ''[http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler PerlLogHandler]'' pragma e.g.

	~~'''~~PerlLogHandler EPrints::Apache::LogHandler~~'''~~		<code>PerlLogHandler EPrints::Apache::LogHandler</code>

	This handler is called after a response has been generated by the web server (i.e. in response to requests coming in from web clients).		This handler is called after a response has been generated by the web server (i.e. in response to requests coming in from web clients).
Line 70:		Line 68:
	==Exposing Usage==		==Exposing Usage==

	Eprints 3 includes a ~~disabled~~ script that can expose access logs through an [http://www.openarchives.org/ OAI-PMH] interface. This interface is located at <tt>/cgi/oai_accesslogs</tt>. The interface contains an <code>EPrints::abort</code> at the top that prevents all access - this is because usage logs are potentially sensitive data~~, and~~ a repository will ~~probably~~ want to implement some kind of access restriction before providing access to their logs.		Eprints 3 includes a script that can expose access logs through an [http://www.openarchives.org/ OAI-PMH] interface. This interface is located at <tt>/cgi/oai_accesslogs</tt>. The interface contains an <code>EPrints::abort</code> at the top that prevents all access - this is because usage logs are potentially sensitive data: a repository will want to implement some kind of access restriction before providing access to their logs.

	The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) to XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).		The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) to XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).
Line 97:		Line 95:
	\|}		\|}

	A result set of records is generated ~~from~~ the OAI ~~request~~. Each record is converted to XML, depending on the export plugin requested (via a mapping of metadataPrefix to plugin identifier), and output as an XML document.		An OAI ListRecords request may contain date and set filters, although sets aren't yet supported for access logs. Using the Eprints search API a result set of records is generated that matches the OAI arguments e.g. all access log records made after a certain time. Each record is converted to XML, depending on the export plugin requested (via a mapping of the metadataPrefix OAI argument to plugin identifier), and output as an XML document.

			Currently Eprints supports exporting access log records in OpenURL ContextObject format. To be OAI-compliant an interface should also support Dublin Core (but access logs don't make a lot of sense in DC). The following is our example request, exposed as an XML ContextObject in an OAI response:

	~~Currently Eprints supports exporting access log records in OpenURL ContextObject format~~. ~~To be OAI~~-~~compliant an interface should also support Dublin Core~~ (~~but access logs don't make a lot of sense in DC~~).		<pre>
			<ctx:context-object timestamp="2007-05-25T13:07:19Z" >
			<ctx:referent>
			<ctx:identifier>oai:eprints.ecs.soton.ac.uk:12614</ctx:identifier>
			</ctx:referent>
			<ctx:referring-entity>
			<ctx:identifier>http://www.w3.org/2001/sw/</ctx:identifier>
			</ctx:referring-entity>
			<ctx:requester>
			<ctx:identifier>urn:ip:127.0.0.1</ctx:identifier>
			<ctx:private-data>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)</ctx:private-data>
			</ctx:requester>
			<ctx:service-type>
			<ctx:metadata-by-val>
			<ctx:format>info:ofi/fmt:xml:xsd:sch_svc</ctx:format>
			<sv:svc-list>
			<sv:fulltext>yes</sv:fulltext>
			</sv:svc-list>
			</ctx:metadata-by-val>
			</ctx:service-type>
			</ctx:context-object>
			</pre>

	==Analysing Usage==		==Analysing Usage==

	~~=Analytical Considerations for Usage=~~		TODO: epstats

Timbrody at 12:57, 25 May 2007

2007-05-25T12:57:10Z

← Older revision		Revision as of 12:57, 25 May 2007
Line 20:		Line 20:

	{\| class="wikitable"		{\| class="wikitable"
			\|+Fields in a HTTP request.
	\|-		\|-
	!field		!field
Line 43:		Line 44:
	\|}		\|}

	In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. Requests are then split up into three sets, non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'':		In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. Requests are then split up into three sets, non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'' field.

	{\| class="wikitable"		{\| class="wikitable"
			\|+Parts of a page request URL.
	\|-		\|-
	!/12614/		!/12614/
Line 51:		Line 53:
	!Semantic_Web_Revisted.pdf		!Semantic_Web_Revisted.pdf
	\|-		\|-
	\|eprint identifier		\|align="center" \|eprint identifier
	\|document identifier		\|align="center" \|document identifier
	\|file requested		\|align="center" \|file requested
	\|}		\|}

			Request without an eprint identifier part are ignored. Requests with no document identifier part are defined as abstract requests. Requests with both eprint and document identifier parts are defined as full-text requests.

			A simple filter is implemented in the capture code to handle requests to multi-file documents (e.g. HTML pages with inlined images). Any full-text requests where the referring web page (see above) is also a request to the same full-text are ignored. Assuming the referring web page is correctly supplied by the web client this will count requests to any part of a document, but ignore multiple requests resulting from complex media.

	==Storing Usage==		==Storing Usage==

			Requests are stored in the Eprints database in the ''access_log'' table, see [Access_Log] for a description of the fields. This is almost a mirror of the normal Apache log file, but by storing the data in a table enables much simpler exposure and re-use of the datab.

			Log data can potentially get very large, which may require a different technical approach in future (or periodic triaging of the access log data).

	==Exposing Usage==		==Exposing Usage==

			Eprints 3 includes a disabled script that can expose access logs through an [http://www.openarchives.org/ OAI-PMH] interface. This interface is located at <tt>/cgi/oai_accesslogs</tt>. The interface contains an <code>EPrints::abort</code> at the top that prevents all access - this is because usage logs are potentially sensitive data, and a repository will probably want to implement some kind of access restriction before providing access to their logs.

			The OAI interface uses Eprints export plugins to convert access log objects (as read from the Eprints database) to XML suitable to be embedded in the OAI output. From an Eprints architecture perspective the mechanism to expose access log records is nearly identical to the exposure of eprint records (they are both ''datasets'' in Eprints).

			{\| class="wikitable"
			\|+accesslog/eprint OAI comparison
			!
			!eprint
			!accesslog
			\|-
			\|identifier
			\|oai:<archive name>:<item id>
			\|oai:<archive name>:<item id>
			\|-
			\|export plugins
			\|(predefined list in cfg.d/oai.pl)
			\|can_accept => "list/access"
			\|-
			\|from/until
			\|last_mod field
			\|datestamp field
			\|-
			\|OAI sets
			\|arbitrary fields defined in cfg.d/oai.pl
			\|not-supported
			\|}

			A result set of records is generated from the OAI request. Each record is converted to XML, depending on the export plugin requested (via a mapping of metadataPrefix to plugin identifier), and output as an XML document.

			Currently Eprints supports exporting access log records in OpenURL ContextObject format. To be OAI-compliant an interface should also support Dublin Core (but access logs don't make a lot of sense in DC).

	==Analysing Usage==		==Analysing Usage==

	=Analytical Considerations for Usage=		=Analytical Considerations for Usage=

Timbrody at 12:24, 25 May 2007

2007-05-25T12:24:25Z

New page

=Download Metrics=

=Download Metrics in EPrints=

==Capturing Usage==

Eprints uses the Apache web server and mod_perl (an accelerator for Perl CGI scripts). mod_perl provides hooks to the logging and control features of the Apache server. One of these hooks is to the access logging function. A ''log handler'' is registered with Apache by using the ''[http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler PerlLogHandler]'' pragma e.g.

'''PerlLogHandler EPrints::Apache::LogHandler'''

This handler is called after a response has been generated by the web server (i.e. in response to requests coming in from web clients).

In the initial implementation we are only interested in requests to eprint objects. This ignores requests to static content (e.g. the home page), searches and user-specific content (e.g. depositing processes). This is because our current requirements are to capture usage to determine download metrics for objects contained in a repository, rather than analysing how a repository is used.

For the remainder of this section we will use the following example request (as you would otherwise see in an Apache access log file):

<code>127.0.0.1 - - [25/May/2007:13:07:24 +0100] "GET /12614/01/Semantic_Web_Revisted.pdf HTTP/1.1" 200 130951 "http://www.w3.org/2001/sw/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"</code>

The pertinent parts of this entry are:

{| class="wikitable"
|-
!field
!example
|-
|requesting host
|127.0.0.1
|-
|date/time of request (by server time)
|25/May/2007:13:07:24 +0100
|-
|page requested
|/12614/01/Semantic_Web_Revisted.pdf
|-
|HTTP response code (200 is 'ok')
|200
|-
|referring web page
|http://www.w3.org/2001/sw/
|-
|user's web browser (or web crawler id)
|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
|}

In the Eprints log handler any non-200 ''HTTP response code'' requests are ignored i.e. any errors or redirects. Requests are then split up into three sets, non-eprint requests, eprint abstract requests and full-text requests. This is based on the ''page requested'':

{| class="wikitable"
|-
!/12614/
!/01/
!Semantic_Web_Revisted.pdf
|-
|eprint identifier
|document identifier
|file requested
|}

==Storing Usage==

==Exposing Usage==

==Analysing Usage==

=Analytical Considerations for Usage=